6
$\begingroup$

I'm in blender 2.92 working with the Python API. I know how to get the pixel coordinates of a vertex before refraction, lens distortion, etc.--see e.g. How to find image coordinates of the rendered vertex?. But this doesn't account for refraction or lens distortion, so e.g. if the object is behind glass, the pixel coordinates are wrong, and I can't find a way to correct it. For example, here I implement the usual matrix world conversion to get the bbox of a sphere inside a glass of water:

Labeled object with IOR not equal to 1

And here I do it with the IOR of glass and water set to 1:

Labeled object with IOR=1

The pixel coordinates are only correct when there is no refraction or distortion.

In the forum answer https://blenderartists.org/t/get-pixel-coordinates-from-the-rendered-image/1163622/5, tricotou suggests

A solution could be to do a first render with a refractive index which equals 1.0 (no displacement of the rays) where the 2D position can be computed. Then do the “normal” render with refraction. And then use an Optical Flow algorithm to track the pixel movement between two renders, and get back the final 2D position.

but this seems pretty complicated. I'm wondering if someone has already implemented something like this or has another simpler solution. Thanks!

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Let's say instead of having a checkerboard background, you have a black background. Now, having only your vertex bounce light (and produce red color in the output) you could render an image, which would contain an information in each pixel on how much this vertex influences that pixel. If you're interested in a single area, you could apply some algorithm that uses some threshold (because perhaps there's no completely black pixels in the result) to divide the image to islands, and then sums up the values of pixels in those areas to pick the area that is the brightest/biggest. $\endgroup$
    – Markus von Broady
    Jun 24, 2021 at 11:43
  • 2
    $\begingroup$ Solving this problem is complicated-- solving this problem is what raytracing does, in a very fundamental sense, and you know that raytracing is a slow, iterative process from watching your Cycles renders tick down. The position of your red sphere depends not just on IOR, but on the distances between meshes, distances along at least two different vectors; distances that are different at different parts of the sphere. Your best bet to find the bounding box of the sphere is with image analysis techniques after raytracing. $\endgroup$
    – Nathan
    Jun 24, 2021 at 15:53
  • $\begingroup$ I've considered rendering plain versions of the scene with a small object at a grid of locations and applying some image thresholding method to locate the object (then interpolating to get a new world --> pixel map), and that's the only viable method I've come up with. I do wonder if it would be possible, through a combination of ray_cast and vector math, to reconstruct at least an approximate version of the rendered pixels. See e.g. this forum. $\endgroup$
    – jmetzger
    Jun 25, 2021 at 17:16
  • 2
    $\begingroup$ A guess: docs.opencv.org/4.5.2/dc/dc3/tutorial_py_matcher.html between undistorted a distorted images. $\endgroup$
    – lemon
    Jun 26, 2021 at 8:30
  • 2
    $\begingroup$ @lemon and adding intermediary steps with intermediary IOR values could perhaps allow a greater success rate of this solution. $\endgroup$
    – Markus von Broady
    Jun 26, 2021 at 11:17

1 Answer 1

Reset to default
4
+100
$\begingroup$

a vertex has single camera position if viewed directly, but can indirectly contribute to zero, all, or any pixels, so no, can't be done in general case.

example

example: there are only two spheres in this scene

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1, but notice how clearly you could draw two rectangles, i.imgur.com/9sqTpuo.png so in principle you can generate some information. If you also calculated some kind of weight, as in how much the red sphere contributes to the color of pixels/region, you could perhaps take the highest weight, which in many cases (like the one in the question) would be the intended result. $\endgroup$
    – Markus von Broady
    Jun 24, 2021 at 7:32
  • $\begingroup$ yeah, for some images, you can get something useful, but here glass has roughness 0, and also floor is lit by spheres $\endgroup$
    – Jordan Szubert
    Jun 24, 2021 at 10:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .