For a research project I am using Blender's Python API to generate some simple random scenes. In addition to the generated renders, I need to acquire a 'foreground mask' in screen (image) coordinates of all the objects in the scene. See below for a simple example using one object:
I would like to know the best way of approaching this problem using the Python API. I several helpful discussions:
- How to find image coordinates of the rendered vertex?
- How to convert coordinates from vertex to world-space?
- https://blenderartists.org/t/3d-coordinates-to-2d-coordinates-in-camera-coordinates-pixels-not-screen-region/561513
Using these sources I have written some code that extracts the object's vertices and convert them to screen coordinates:
world_matrix = obj.matrix_world
modelview_matrix = camera_obj.matrix_world.inverted() # currently not used
projection_matrix = camera_obj.calc_matrix_camera( # currently not used
scene.render.resolution_x,
scene.render.resolution_y,
scene.render.pixel_aspect_x,
scene.render.pixel_aspect_y,
)
render_scale = scene.render.resolution_percentage/100
render_size = (int(scene.render.resolution_x * render_scale),
int(scene.render.resolution_y * render_scale))
for v in obj.data.vertices:
loc = world_matrix * v.co
screen_coords = bpy_extras.object_utils.world_to_camera_view(scene, camera_obj, loc)
pixel_coords = (round(screen_coords.x * render_size[0]),
round(screen_coords.y * render_size[1]))
However, this gives incorrect results but I am not sure what goes wrong. Here is the corresponding output with the yellow points indicating the valid screen coordinates from the code above:
Besides being incorrect, the logic to check whether a vertex is actually in the field of view is missing. I am wondering whether there are some built-in functions that I can use to achieve my goals. For example, I found that Blender Game Engine has the function bge.types.KX_Camera.getScreenPosition
which looks like what I need. Unfortunately, it seems like I cannot import bge
when using Blender in python-only mode without IDE. Another possibility may be bpy_extras.view3d_utils.location_3d_to_region_2d
.
EDIT: I modified the code a little to position the camera in the origin (0,0,0) and then the above code gives the correct result when the object's scale is (1.0,1.0,1.0) and rotation_euler is (0,0,0). However, when I scale or rotate the image the above code does not return correct results. What is the correct way of converting the NDC coordinates to device coordinates? Below is are the example outputs with object's scale 1.0 and 1.2 (full modified code can be found here). I am also still wondering whether this is the way to go, for a simple cube this may work but for more complex meshes this method may become cumbersome.
Any pointers in the right direction are highly appreciated!
- bge.types.KX_Camera.getScreenPosition: https://docs.blender.org/api/2.78a/bge.types.KX_Camera.html#bge.types.KX_Camera.getScreenPosition
- bpy_extras.view3d_utils.location_3d_to_region_2d: https://docs.blender.org/api/blender_python_api_2_75_3/bpy_extras.view3d_utils.html?highlight=3d%20view
bpy_extras.view3d_utils.location_3d_to_region_2d
? Recommend usingfrom bpy_extras.view3d_uitls import location_3d_to_region_3d
. Be very surprised if it is missing from blender. $\endgroup$