I'm in blender 2.92 working with the Python API. I know how to get the pixel coordinates of a vertex before refraction, lens distortion, etc.--see e.g. How to find image coordinates of the rendered vertex?. But this doesn't account for refraction or lens distortion, so e.g. if the object is behind glass, the pixel coordinates are wrong, and I can't find a way to correct it. For example, here I implement the usual matrix world conversion to get the bbox of a sphere inside a glass of water:
And here I do it with the IOR of glass and water set to 1:
The pixel coordinates are only correct when there is no refraction or distortion.
In the forum answer https://blenderartists.org/t/get-pixel-coordinates-from-the-rendered-image/1163622/5, tricotou suggests
A solution could be to do a first render with a refractive index which equals 1.0 (no displacement of the rays) where the 2D position can be computed. Then do the “normal” render with refraction. And then use an Optical Flow algorithm to track the pixel movement between two renders, and get back the final 2D position.
but this seems pretty complicated. I'm wondering if someone has already implemented something like this or has another simpler solution. Thanks!