For a research project I am using Blender's Python API to generate some simple random scenes. In addition to the generated renders, I need to acquire a 'foreground mask' in screen (image) coordinates of all the objects in the scene. See below for a simple example using one object:
I would like to know the best way of approaching this problem using the Python API. I several helpful discussions:
- How to find image coordinates of the rendered vertex?
- How to convert coordinates from vertex to world-space?
Using these sources I have written some code that extracts the object's vertices and convert them to screen coordinates:
world_matrix = obj.matrix_world modelview_matrix = camera_obj.matrix_world.inverted() # currently not used projection_matrix = camera_obj.calc_matrix_camera( # currently not used scene.render.resolution_x, scene.render.resolution_y, scene.render.pixel_aspect_x, scene.render.pixel_aspect_y, ) render_scale = scene.render.resolution_percentage/100 render_size = (int(scene.render.resolution_x * render_scale), int(scene.render.resolution_y * render_scale)) for v in obj.data.vertices: loc = world_matrix * v.co screen_coords = bpy_extras.object_utils.world_to_camera_view(scene, camera_obj, loc) pixel_coords = (round(screen_coords.x * render_size), round(screen_coords.y * render_size))
However, this gives incorrect results but I am not sure what goes wrong. Here is the corresponding output with the yellow points indicating the valid screen coordinates from the code above:
Besides being incorrect, the logic to check whether a vertex is actually in the field of view is missing. I am wondering whether there are some built-in functions that I can use to achieve my goals. For example, I found that Blender Game Engine has the function
bge.types.KX_Camera.getScreenPosition which looks like what I need. Unfortunately, it seems like I cannot
import bge when using Blender in python-only mode without IDE. Another possibility may be
EDIT: I modified the code a little to position the camera in the origin (0,0,0) and then the above code gives the correct result when the object's scale is (1.0,1.0,1.0) and rotation_euler is (0,0,0). However, when I scale or rotate the image the above code does not return correct results. What is the correct way of converting the NDC coordinates to device coordinates? Below is are the example outputs with object's scale 1.0 and 1.2 (full modified code can be found here). I am also still wondering whether this is the way to go, for a simple cube this may work but for more complex meshes this method may become cumbersome.
Any pointers in the right direction are highly appreciated!
- bge.types.KX_Camera.getScreenPosition: https://docs.blender.org/api/2.78a/bge.types.KX_Camera.html#bge.types.KX_Camera.getScreenPosition
- bpy_extras.view3d_utils.location_3d_to_region_2d: https://docs.blender.org/api/blender_python_api_2_75_3/bpy_extras.view3d_utils.html?highlight=3d%20view