Compute object's foreground mask in screen coordinates

Question

For a research project I am using Blender's Python API to generate some simple random scenes. In addition to the generated renders, I need to acquire a 'foreground mask' in screen (image) coordinates of all the objects in the scene. See below for a simple example using one object:

I would like to know the best way of approaching this problem using the Python API. I several helpful discussions:

• How to find image coordinates of the rendered vertex?
• How to convert coordinates from vertex to world-space?
• https://blenderartists.org/t/3d-coordinates-to-2d-coordinates-in-camera-coordinates-pixels-not-screen-region/561513

Using these sources I have written some code that extracts the object's vertices and convert them to screen coordinates:

world_matrix = obj.matrix_world
modelview_matrix  = camera_obj.matrix_world.inverted()   # currently not used
projection_matrix = camera_obj.calc_matrix_camera(       # currently not used
    scene.render.resolution_x,
    scene.render.resolution_y,
    scene.render.pixel_aspect_x,
    scene.render.pixel_aspect_y,
)

render_scale = scene.render.resolution_percentage/100
render_size = (int(scene.render.resolution_x * render_scale), 
               int(scene.render.resolution_y * render_scale))

for v in obj.data.vertices:
    loc = world_matrix * v.co
    screen_coords = bpy_extras.object_utils.world_to_camera_view(scene, camera_obj, loc)
    pixel_coords = (round(screen_coords.x * render_size[0]), 
                    round(screen_coords.y * render_size[1]))

However, this gives incorrect results but I am not sure what goes wrong. Here is the corresponding output with the yellow points indicating the valid screen coordinates from the code above:

Besides being incorrect, the logic to check whether a vertex is actually in the field of view is missing. I am wondering whether there are some built-in functions that I can use to achieve my goals. For example, I found that Blender Game Engine has the function bge.types.KX_Camera.getScreenPosition which looks like what I need. Unfortunately, it seems like I cannot import bge when using Blender in python-only mode without IDE. Another possibility may be bpy_extras.view3d_utils.location_3d_to_region_2d.

EDIT: I modified the code a little to position the camera in the origin (0,0,0) and then the above code gives the correct result when the object's scale is (1.0,1.0,1.0) and rotation_euler is (0,0,0). However, when I scale or rotate the image the above code does not return correct results. What is the correct way of converting the NDC coordinates to device coordinates? Below is are the example outputs with object's scale 1.0 and 1.2 (full modified code can be found here). I am also still wondering whether this is the way to go, for a simple cube this may work but for more complex meshes this method may become cumbersome.

Any pointers in the right direction are highly appreciated!

• bge.types.KX_Camera.getScreenPosition: https://docs.blender.org/api/2.78a/bge.types.KX_Camera.html#bge.types.KX_Camera.getScreenPosition
• bpy_extras.view3d_utils.location_3d_to_region_2d: https://docs.blender.org/api/blender_python_api_2_75_3/bpy_extras.view3d_utils.html?highlight=3d%20view

Answer 1

I was not aware of Blender's ability to render the scene depth map. This is exactly what I need to obtain the foreground mask. The code snippet at the link below produces the desired results, specifically when setting the map.size value to a very low number, e.g. 0.001.

Render depth map to image with python script