How to find image coordinates of the rendered vertex?

Question

I have a simple mesh of a cube. I would like to know what are the coordinates of each vertex in the rendered image (i.e. for each 3d vertex I'd like to know it's pixel coordinates). Is there a way to do this?

Clarification:

I'm experimenting with a pose estimation algorithm (i.e. finding out the 3d pose of the object) from a (set of) 2d image(s). Instead of taking an image of a real object with a real camera I synthetically render the image of a cube using blender. In this way I know the pose of the object and the intrinsics of the camera, so I can easily validate the results of the pose estimation algorithm. Pose estimation algorithms usually require to provide them with world points on the object and the corresponding image points. So what I'm looking for is a way to produce a dump of correspondences of the 3d point <--> image point as a byproduct of the rendering procedure. Thanks, Alex

Answer 1

This script gives projected the 2D pixel coordinate from a 3D world-space location.

• The (0, 0) origin is the bottom left of the image
  (common for math, OpenGL and Blender in general). note - cursor_location should read cursor.location
• A negative Z return value means the point is behind the camera:

Example:

# Test the function using the active object (which must be a camera)
# and the 3D cursor as the location to find.
import bpy
import bpy_extras

scene = bpy.context.scene
obj = bpy.context.object
co = bpy.context.scene.cursor.location

co_2d = bpy_extras.object_utils.world_to_camera_view(scene, obj, co)
print("2D Coords:", co_2d)

# If you want pixel coords
render_scale = scene.render.resolution_percentage / 100
render_size = (
    int(scene.render.resolution_x * render_scale),
    int(scene.render.resolution_y * render_scale),
)
print("Pixel Coords:", (
      round(co_2d.x * render_size[0]),
      round(co_2d.y * render_size[1]),
))

See the Python Source-Code if you're interested in exactly how this works.

Answer 2

The new world_to_camera_view function returns x, y, z-depth.

This will print out the 2d coordinate list, (no z-depth)

Example 1

import bpy
from bpy_extras.object_utils import world_to_camera_view

scene = bpy.context.scene

# needed to rescale 2d coordinates
render = scene.render
res_x = render.resolution_x
res_y = render.resolution_y

obj = bpy.data.objects['Cube']
cam = bpy.data.objects['Camera']

# use generator expressions () or list comprehensions []
verts = (vert.co for vert in obj.data.vertices)
coords_2d = [world_to_camera_view(scene, cam, coord) for coord in verts]

# 2d data printout:
rnd = lambda i: round(i)

print('x,y')
for x, y, distance_to_lens in coords_2d:
    print("{},{}".format(rnd(res_x*x), rnd(res_y*y)))

this:

can become this: (here is a live d3.js rendering using svg - chrome)

Example 2

taking z-depth (distance to lens) into account (skipping code already present above):

verts = (vert.co for vert in obj.data.vertices)
coords_2d = [world_to_camera_view(scene, cam, coord) for coord in verts]

# find min max distance, between eye and coordinate.
rnd = lambda i: round(i)
rnd3 = lambda i: round(i, 3)

limit_finder = lambda f: f(coords_2d, key=lambda i: i[2])[2]
limits = limit_finder(min), limit_finder(max)
limits = [rnd3(d) for d in limits]

print('min, max\n{},{}'.format(*limits))

# x, y, d=distance_to_lens
print('x,y,d')
for x, y, d in coords_2d:
    print("{},{},{}".format(rnd(res_x*x), rnd(res_y*y), rnd3(d)))

you can generate this kind of information (here live svg sample again - chrome): Using distance to set fill-opacity of the dots.

Answer 3

I am not very good with python so I will not give some code here, but I will try to cover the theoretical/mathematical part of the question:

These are standard 3D graphics manipulations.

Some more information: Vectors in 3D graphics are 4 dimensional entities. The 3 first coordinates correspond to the euclidean space x,y,z coordinates, usually in local or model space and the last coordinate is the w or homogenous coordinate, which is used for perspective effects, translation in 3d space and for clipping.

Likewise, 3D space matrices are 4x4 matrices, so that they can handle the 4 dimensional vertices. Usually when a vertex is transformed to screen space, the initial value of w is 1.0. There are other values to use but they are for more advanced use.

Transforming to screen space is simply a matter of transforming the vertex by the Model matrix M, to transform the vertex to world space (where the model resides on the 3D world), by the camera matrix C, to transform it to the reference frame of the camera, or camera space, by the projection matrix P to account for projection effects (called projection space), do perspective division and clipping and finally turn the resulting coordinates to pixel space.

So if the vertex is v, you have to do P*C*M*v (remember matrix multiplications are non-commutative! Order of multiplication is important!). This has transformed the vector to projection space

After that, you need to clip the vector. If any of the vector's 3 first components is greater in absolute value than the fourth component, then it is outside the field of view of the camera and should be clipped. If the vector is not clipped, then you need to divide each of the 3 components of the vector by the w coordinate. So if the projection space vector is (x,y,z,w), after clipping you get the screen space coordinates (x/w, y/w, z/w, 1)

This will yield normalized coordinates with range {-1.0, 1.0}. To convert to pixel space, you need to know the width and height of the screen in pixels.

"Screen" here, may not refer to an actual screen, but can be the rendered image. What changes in the description is that instead of an actual screen width/height, you have the x/y resolution of the rendered image.

If (sx, sy) are the screen space vector x and y coordinates and w, h are the height and width of the screen, or image, then

x = w/2 * (1.0 + sx) will give the x position in screen space.

For the y component it may be a little more complicated because usually pixel space is counted from the bottom of the screen up, while operating systems count it on reverse. So, depending on the occasion it may be either

y = h/2 * (1.0 + sy) or
y = h/2 * (1.0 - sy)

A warning though, the names of the spaces can be different in some textbooks. I hope that I have managed to relay the general idea though.

For more information I recommend reading

http://www.glprogramming.com/red/appendixf.html

Answer 4

Followed the instructions from this site: songho.ca/opengl/gl_projectionmatrix
There is also a course available at udacity which provides further informations.

The result looks like this.

import bpy
from math import tan
from mathutils import Matrix

print(79*'-' + 3*'\n')

cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']
scene = bpy.context.scene
mesh = obj.data
model_view = (
    cam.matrix_world.inverted() * 
    obj.matrix_world
    )

width =  scene.render.resolution_x 
height = scene.render.resolution_y
aspect_ratio = width / height

n = cam.data.clip_start
f = cam.data.clip_end
fov = cam.data.angle

proj = Matrix()
proj[0][0] =            1 / tan(fov / 2)
proj[1][1] = aspect_ratio / tan(fov / 2)
proj[2][2] = -(f + n) / (f - n)
proj[2][3] = - 2*f*n  / (f - n)
proj[3][2] = - 1
proj[3][3] =   0

clip = proj * model_view

for v in mesh.vertices:
    v_4d = v.co.copy()
    v_4d.resize_4d()

    v_clip  = clip * v_4d
    v_clip /= v_clip[3]
    v.co = v_clip.resized(3)

    scrn_co_x = (v.co.x + 1) / 2 * width
    scrn_co_y = (v.co.y + 1) / 2 * height
    print("screen: %.2f, %.2f" % (scrn_co_x, scrn_co_y))

Answer 5

An alternative way to @ideasman42's answer is to first create a 3x4 P matrix (as used in computer vision) from blender's camera parameters and use that to project. This might be convenient for you since you seem to be doing computer vision work.

Use the function get_3x4_RT_matrix_from_blender listed in my answer to 3x4 camera matrix from blender camera to generate a P matrix, then project. For instance:

# Insert your camera name here
cam = bpy.data.objects['Camera.001']
P = get_3x4_P_matrix_from_blender(cam)
e1 = Vector((1, 0,    0, 1))
p1 = P * e1
p1 /= p1[2]
print("Projected e1")
print(p1)

For vertices in the object's local coordinates, you will have to first transform them to world coordinates. Here's a sketch of how to do it:

location, rotation, scale = obj.matrix_world.decompose()
R = rotation.to_matrix().transposed();
p = R*scale*vertex_local_coordinates + location
# now apend coordinate 1 to p, use P*p then divide by 3rd coordinate

You can compare that to @ideasman42's code which I made into a function project_by_object_utils(cam, point) (see my answer to 3x4 camera matrix from blender camera). Both routines return exactly the same coordinates up to many decimal places.

• Note: The origin is at the top-left corner and image y axis pointing down.

Answer 6

the last answer with the less likes, from Shajahan Kabir Sajuu, is awsome.
I will simplify the problem around selection of vertices: front vertices and not back ones. the problem when using only "world_to_camera_view" I imagined the same way of adding a raycast but his idea to use location with a little object created at every vertex is amazing. I extended his script to several selected objects and with depsgraph to take into account geometry when modifiers not applied.

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view

def main(context):
    # Get context elements: scene, camera and mesh
    scene = context.scene
    cam = context.scene.camera
    objs = context.selected_objects
    cao=context.active_object       ##to restore it at the end

    limit = 0.1

    # In world coordinates, get a bvh tree and vertices
    bpy.ops.object.mode_set(mode='OBJECT')
    for obj in objs:   
        if obj.type=='MESH':
            me = obj.data
            depsgraph = context.evaluated_depsgraph_get()
            obj_eval = obj.evaluated_get(depsgraph)
            for p in me.polygons:     #deselection of vertices (face then edges)
                p.select = False
            for e in me.edges:
                e.select = False         

            for v in me.vertices:

                mw = obj_eval.matrix_world  #matrix on the obj_eval of the depsgraph

                # Get the 2D projection of the vertex
                co2D = world_to_camera_view( scene, cam, mw @ v.co )

                cubeMesh=bpy.ops.mesh.primitive_cube_add(location=(mw @ v.co)) 
                bpy.ops.transform.resize(value=(0.01, 0.01, 0.01))
                cubeObject=context.object

                # If inside the camera view
                if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0 and co2D.z >0: 
                    # Try a ray cast, in order to test the vertex visibility from the camera                    
                    location= scene.ray_cast(context.view_layer, cam.location, (mw @ v.co - cam.location).normalized() )
                    # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
                    print(location[0],location[1]) # first value=bool, second=coords
                    if location[0] and (mw @ v.co - location[1]).length < limit:
                        v.select = True

                bpy.data.objects.remove(cubeObject,do_unlink= True) #delete object
    context.view_layer.objects.active =cao  #restore active
    cao.select_set(True)

so I think he explained it well but without testing this is hard to realize.

Answer 7

This is my code:

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

#renderer an image and save to F drive
scene.render.image_settings.file_format = 'PNG'
scene.render.filepath = "F:/image.png"
bpy.ops.render.render(write_still = 1)

width =  scene.render.resolution_x 
height = scene.render.resolution_y

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.1

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get a bvh tree and vertices
mWorld = obj.matrix_world
vertices = [mWorld @ v.co for v in obj.data.vertices]

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    cubeMesh=bpy.ops.mesh.primitive_cube_add(location=(v))
    bpy.ops.transform.resize(value=(0.01, 0.01, 0.01))
    cubeObject=bpy.context.object

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0 and co2D.z >0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        location= scene.ray_cast(bpy.context.view_layer, cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if location[0] and (v - location[1]).length < limit:
            print("%.2f %.2f" %(co2D.x*width,co2D.y*height))
    bpy.data.objects.remove(cubeObject,do_unlink= True)