2
$\begingroup$

I have 3D model of a scene with several objects that are transformed from one keyframe to another one (translation, rotation, scaling). This scene will be rendered as animation with write_still=True so the final results are images per each frame. I am interested in finding pixels that are visible in one frame but are occluded in the next frame. A simple example could be something like this:

First frame ----------- > Second frame

enter image description here enter image description here

In this example, the colorful plane is fully visible in the first frame but is getting behind the gray square in the second frame, so not only the bottom left corner of it is occluded now, but part of the background near the gray square is also occluded by the colorful square. In the next image, the occluded areas are highlighted white. I would like to have a binary mask that shows which pixels from the first frame will be occluded in the next frame. The final goal is to do this by Python interface, but any suggestion to solve this is very appreciated.

Desired occlusion map

Please note that objects are 3D, they might rotate or scale too. Lighting will change and the camera motion is also involved. A more complete example is as follows:

First frame

enter image description here

Second frame

enter image description here

Occlusion map

enter image description here

Improve this question
$\endgroup$
9
  • $\begingroup$ The difference of two images (image1 - image2) is supported in the Compositor of Blender. Images can be compared by the standard math operations less >, greater <, equal == et cetera.Then you can threshold the result by various means including a color ramp (mapping color) . Perhaps this is a starting point. You may want to search the Blender API for image functions. $\endgroup$
    – atomicbezierslinger
    Apr 22, 2020 at 4:54
  • $\begingroup$ What is the larger context of you work? Is the work post production, part of a pipeline, ...? Realtime? If you are doing something very elaborate please state so if possible. $\endgroup$
    – atomicbezierslinger
    Apr 22, 2020 at 5:00
  • $\begingroup$ Thank you @atomicbezierslinger for this point, but actually, in a more realistic example, differencing images won't work. The camera will move around, lighting will change, etc. I need to kind of keep track of visible vertices to the camera or something! $\endgroup$
    – Ali Salehi
    Apr 22, 2020 at 5:03
  • $\begingroup$ Using the speed vectors, I am finding the motion vector for each pixel between frames, now I need to determine which pixels are not visible in the second frame. $\endgroup$
    – Ali Salehi
    Apr 22, 2020 at 5:08
  • 1
    $\begingroup$ Again the compositor may help. To a large degree you can use Object ID ,a number say, for each object, lets say N1 and N2. Then the pixel itself can be examined for those IDs, N1 and N2. The object ID is a feature of Blender. You may want to add your additional variations to the questions itself. $\endgroup$
    – atomicbezierslinger
    Apr 22, 2020 at 5:17

1 Answer 1

Reset to default
-2
$\begingroup$

enter image description here

Please see the panel ... object relations pass index. This information can be examined in the compositor. This is generated in the object index render pass for Cycles. You must enable that pass.

enter image description here

There is also Cryptomatte.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could you please explain more? I am aware of the Object Index Pass, but how can that be useful here? Imagine a sphere rotating in place, the back of it coming to view in the next frame while its front pixels are getting occluded. Object index will be the same for these pixels in both frames, but we know that all those pixels should be considered as occlusion. $\endgroup$
    – Ali Salehi
    Apr 22, 2020 at 16:23
  • $\begingroup$ You might consider examining the geometric normals of faces in addtion to object id. You might also add the refinements written immediately above to your original question. There is a language problem of speaking about faces versus pixels. There is also an issue of how exact you are trying to be. I might not be able produce any more suggestions. You might consider rendering with materials designed to be trackable via object id or material id. I do not know the larger goals or your work, so I cannot make many suggestion with a rationale. I am guessing. $\endgroup$
    – atomicbezierslinger
    Apr 22, 2020 at 17:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .