Finding image coordinates of rendered vertex after refraction, distortion, etc

I'm in blender 2.92 working with the Python API. I know how to get the pixel coordinates of a vertex before refraction, lens distortion, etc.--see e.g. How to find image coordinates of the rendered vertex?. But this doesn't account for refraction or lens distortion, so e.g. if the object is behind glass, the pixel coordinates are wrong, and I can't find a way to correct it. For example, here I implement the usual matrix world conversion to get the bbox of a sphere inside a glass of water:

And here I do it with the IOR of glass and water set to 1:

The pixel coordinates are only correct when there is no refraction or distortion.

In the forum answer https://blenderartists.org/t/get-pixel-coordinates-from-the-rendered-image/1163622/5, tricotou suggests

A solution could be to do a first render with a refractive index which equals 1.0 (no displacement of the rays) where the 2D position can be computed. Then do the “normal” render with refraction. And then use an Optical Flow algorithm to track the pixel movement between two renders, and get back the final 2D position.

but this seems pretty complicated. I'm wondering if someone has already implemented something like this or has another simpler solution. Thanks!

• Let's say instead of having a checkerboard background, you have a black background. Now, having only your vertex bounce light (and produce red color in the output) you could render an image, which would contain an information in each pixel on how much this vertex influences that pixel. If you're interested in a single area, you could apply some algorithm that uses some threshold (because perhaps there's no completely black pixels in the result) to divide the image to islands, and then sums up the values of pixels in those areas to pick the area that is the brightest/biggest. Jun 24, 2021 at 11:43
• Solving this problem is complicated-- solving this problem is what raytracing does, in a very fundamental sense, and you know that raytracing is a slow, iterative process from watching your Cycles renders tick down. The position of your red sphere depends not just on IOR, but on the distances between meshes, distances along at least two different vectors; distances that are different at different parts of the sphere. Your best bet to find the bounding box of the sphere is with image analysis techniques after raytracing. Jun 24, 2021 at 15:53
• I've considered rendering plain versions of the scene with a small object at a grid of locations and applying some image thresholding method to locate the object (then interpolating to get a new world --> pixel map), and that's the only viable method I've come up with. I do wonder if it would be possible, through a combination of ray_cast and vector math, to reconstruct at least an approximate version of the rendered pixels. See e.g. this forum. Jun 25, 2021 at 17:16
• A guess: docs.opencv.org/4.5.2/dc/dc3/tutorial_py_matcher.html between undistorted a distorted images. Jun 26, 2021 at 8:30
• @lemon and adding intermediary steps with intermediary IOR values could perhaps allow a greater success rate of this solution. Jun 26, 2021 at 11:17

1 Answer

a vertex has single camera position if viewed directly, but can indirectly contribute to zero, all, or any pixels, so no, can't be done in general case.

example: there are only two spheres in this scene

• +1, but notice how clearly you could draw two rectangles, i.imgur.com/9sqTpuo.png so in principle you can generate some information. If you also calculated some kind of weight, as in how much the red sphere contributes to the color of pixels/region, you could perhaps take the highest weight, which in many cases (like the one in the question) would be the intended result. Jun 24, 2021 at 7:32
• yeah, for some images, you can get something useful, but here glass has roughness 0, and also floor is lit by spheres Jun 24, 2021 at 10:11