29
$\begingroup$

I have a simple mesh of a cube. I would like to know what are the coordinates of each vertex in the rendered image (i.e. for each 3d vertex I'd like to know it's pixel coordinates). Is there a way to do this?

Clarification:

I'm experimenting with a pose estimation algorithm (i.e. finding out the 3d pose of the object) from a (set of) 2d image(s). Instead of taking an image of a real object with a real camera I synthetically render the image of a cube using blender. In this way I know the pose of the object and the intrinsics of the camera, so I can easily validate the results of the pose estimation algorithm. Pose estimation algorithms usually require to provide them with world points on the object and the corresponding image points. So what I'm looking for is a way to produce a dump of correspondences of the 3d point <--> image point as a byproduct of the rendering procedure. Thanks, Alex

$\endgroup$
  • 2
    $\begingroup$ There are many ways to do this, and the answer depends on the context you are working in, so this question is too vague, you need to explain the intended purpose, I could show the C code which calculates this but I think its probably not what you want. Please give details on the format you want this data in, is a Python script that calculates this acceptable for example or do you need this for a shader, if so OSL or GLSL, or is the game engine function for this enough, do you want to know the math to calculate or just a method to do it for you... etc. $\endgroup$ – ideasman42 Jun 11 '13 at 22:39
  • $\begingroup$ I've updated the question with the clarification note, please see if it's better now. $\endgroup$ – Alex Kreimer Jun 12 '13 at 7:19
  • 1
    $\begingroup$ You still didn't give any info on the context language and APIs you expect to be using. $\endgroup$ – ideasman42 Jun 12 '13 at 23:18
  • 2
    $\begingroup$ C++/Python are fine. I'd expect the API to be Blenders' one... $\endgroup$ – Alex Kreimer Jun 13 '13 at 4:58
29
$\begingroup$

This script gives projected the 2D pixel coordinate from a 3D world-space location.

  • The (0, 0) origin is the bottom left of the image
    (common for math, OpenGL and Blender in general).
  • A negative Z return value means the point is behind the camera:

Example:

# Test the function using the active object (which must be a camera)
# and the 3D cursor as the location to find.
import bpy
import bpy_extras

scene = bpy.context.scene
obj = bpy.context.object
co = bpy.context.scene.cursor_location

co_2d = bpy_extras.object_utils.world_to_camera_view(scene, obj, co)
print("2D Coords:", co_2d)

# If you want pixel coords
render_scale = scene.render.resolution_percentage / 100
render_size = (
        int(scene.render.resolution_x * render_scale),
        int(scene.render.resolution_y * render_scale),
        )
print("Pixel Coords:", (
      round(co_2d.x * render_size[0]),
      round(co_2d.y * render_size[1]),
      ))

See the Python Source-Code if you're interested in exactly how this works.

$\endgroup$
  • 4
    $\begingroup$ This code works, but keep in mind it returns pixel (u,v) coordinates with the v axis pointing up and the origin at the bottom-left corner. If you want the (i,j) coordinates as in a matrix, with the origin on the top-left corner and i axis pointing down, do i = render_size[1] - v and j = u. I've tested this and it works, with no off-by-one errors. $\endgroup$ – rfabbri Sep 7 '15 at 23:31
  • 1
    $\begingroup$ @rfabbri, 0,0 for lower left is used everywhere in Blender & OpenGL, so it didnt cross my mind to mention it. Edited answer. $\endgroup$ – ideasman42 Sep 8 '15 at 3:16
  • $\begingroup$ When we have non-zero camera shift_x/y, this code no longer works. I am working with blender 2.79 $\endgroup$ – iNFINITEi Nov 2 '17 at 3:20
  • $\begingroup$ @iNFINITEi report to Blender's bug tracker $\endgroup$ – ideasman42 Nov 2 '17 at 9:21
  • $\begingroup$ Can we also know whether the vertex is visible (blocked by some surfaces) or not given specific camera view? $\endgroup$ – Jon Aug 14 at 10:58
10
$\begingroup$

I am not very good with python so I will not give some code here, but I will try to cover the theoretical/mathematical part of the question:

These are standard 3D graphics manipulations.

Some more information: Vectors in 3D graphics are 4 dimensional entities. The 3 first coordinates correspond to the euclidean space x,y,z coordinates, usually in local or model space and the last coordinate is the w or homogenous coordinate, which is used for perspective effects, translation in 3d space and for clipping.

Likewise, 3D space matrices are 4x4 matrices, so that they can handle the 4 dimensional vertices. Usually when a vertex is transformed to screen space, the initial value of w is 1.0. There are other values to use but they are for more advanced use.

Transforming to screen space is simply a matter of transforming the vertex by the Model matrix M, to transform the vertex to world space (where the model resides on the 3D world), by the camera matrix C, to transform it to the reference frame of the camera, or camera space, by the projection matrix P to account for projection effects (called projection space), do perspective division and clipping and finally turn the resulting coordinates to pixel space.

So if the vertex is v, you have to do P*C*M*v (remember matrix multiplications are non-commutative! Order of multiplication is important!). This has transformed the vector to projection space

After that, you need to clip the vector. If any of the vector's 3 first components is greater in absolute value than the fourth component, then it is outside the field of view of the camera and should be clipped. If the vector is not clipped, then you need to divide each of the 3 components of the vector by the w coordinate. So if the projection space vector is (x,y,z,w), after clipping you get the screen space coordinates (x/w, y/w, z/w, 1)

This will yield normalized coordinates with range {-1.0, 1.0}. To convert to pixel space, you need to know the width and height of the screen in pixels.

"Screen" here, may not refer to an actual screen, but can be the rendered image. What changes in the description is that instead of an actual screen width/height, you have the x/y resolution of the rendered image.

If (sx, sy) are the screen space vector x and y coordinates and w, h are the height and width of the screen, or image, then

x = w/2 * (1.0 + sx) will give the x position in screen space.

For the y component it may be a little more complicated because usually pixel space is counted from the bottom of the screen up, while operating systems count it on reverse. So, depending on the occasion it may be either

y = h/2 * (1.0 + sy) or
y = h/2 * (1.0 - sy)

A warning though, the names of the spaces can be different in some textbooks. I hope that I have managed to relay the general idea though.

For more information I recommend reading

http://www.glprogramming.com/red/appendixf.html

$\endgroup$
  • $\begingroup$ Thanks, it's a nice explanation. Please note that it's rather unrelated to the question that was asked. $\endgroup$ – Alex Kreimer Jun 11 '13 at 9:48
  • $\begingroup$ Not at all. The principles are exactly the same, you just substitute the width of the screen with the image dimensions. I will update the answer $\endgroup$ – Antony Riakiotakis Jun 11 '13 at 9:51
  • 1
    $\begingroup$ I realize that your answer is relevant in general. Having said that, please note that I'm looking for a way (using python api, as you note) to ask blender what are the projected image coordinates. Thanks for your effort though. $\endgroup$ – Alex Kreimer Jun 11 '13 at 11:23
  • $\begingroup$ How does this account for occlusion of a point by a surface in front of the point? If the face occludes a point and the camera i.e a point is behind a surface then it won't be visible in the image. Ideally, there should be no corresponding image coordinate if it has been occluded. Is there a way to get this? $\endgroup$ – web_ninja Feb 19 '16 at 0:29
8
$\begingroup$

The new world_to_camera_view function returns x, y, z-depth.

This will print out the 2d coordinate list, (no z-depth)

Example 1

import bpy
from bpy_extras.object_utils import world_to_camera_view

scene = bpy.context.scene

# needed to rescale 2d coordinates
render = scene.render
res_x = render.resolution_x
res_y = render.resolution_y

obj = bpy.data.objects['Cube']
cam = bpy.data.objects['Camera']

# use generator expressions () or list comprehensions []
verts = (vert.co for vert in obj.data.vertices)
coords_2d = [world_to_camera_view(scene, cam, coord) for coord in verts]

# 2d data printout:
rnd = lambda i: round(i)

print('x,y')
for x, y, distance_to_lens in coords_2d:
    print("{},{}".format(rnd(res_x*x), rnd(res_y*y)))

this:

enter image description here

can become this: (here is a live d3.js rendering using svg - chrome)

enter image description here

Example 2

taking z-depth (distance to lens) into account (skipping code already present above):

verts = (vert.co for vert in obj.data.vertices)
coords_2d = [world_to_camera_view(scene, cam, coord) for coord in verts]

# find min max distance, between eye and coordinate.
rnd = lambda i: round(i)
rnd3 = lambda i: round(i, 3)

limit_finder = lambda f: f(coords_2d, key=lambda i: i[2])[2]
limits = limit_finder(min), limit_finder(max)
limits = [rnd3(d) for d in limits]

print('min, max\n{},{}'.format(*limits))

# x, y, d=distance_to_lens
print('x,y,d')
for x, y, d in coords_2d:
    print("{},{},{}".format(rnd(res_x*x), rnd(res_y*y), rnd3(d)))

you can generate this kind of information (here live svg sample again - chrome): Using distance to set fill-opacity of the dots.

enter image description here

$\endgroup$
6
$\begingroup$

Followed the instructions from this site: songho.ca/opengl/gl_projectionmatrix
There is also a course available at udacity which provides further informations.

The result looks like this.

viewport

import bpy
from math import tan
from mathutils import Matrix

print(79*'-' + 3*'\n')

cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']
scene = bpy.context.scene
mesh = obj.data
model_view = (
    cam.matrix_world.inverted() * 
    obj.matrix_world
    )

width =  scene.render.resolution_x 
height = scene.render.resolution_y
aspect_ratio = width / height

n = cam.data.clip_start
f = cam.data.clip_end
fov = cam.data.angle

proj = Matrix()
proj[0][0] =            1 / tan(fov / 2)
proj[1][1] = aspect_ratio / tan(fov / 2)
proj[2][2] = -(f + n) / (f - n)
proj[2][3] = - 2*f*n  / (f - n)
proj[3][2] = - 1
proj[3][3] =   0

clip = proj * model_view

for v in mesh.vertices:
    v_4d = v.co.copy()
    v_4d.resize_4d()

    v_clip  = clip * v_4d
    v_clip /= v_clip[3]
    v.co = v_clip.resized(3)

    scrn_co_x = (v.co.x + 1) / 2 * width
    scrn_co_y = (v.co.y + 1) / 2 * height
    print("screen: %.2f, %.2f" % (scrn_co_x, scrn_co_y))
$\endgroup$
  • $\begingroup$ Coordinates seem to match now, multiplying the y-axis by the aspect ratio instead of the x-axis. $\endgroup$ – pink vertex Nov 29 '14 at 11:57
  • $\begingroup$ I think you mixed up something with the proj indices. When I try your code I get a IndexError: vector[index] = x: assignment index out of range $\endgroup$ – Masala Nov 29 '14 at 16:53
  • $\begingroup$ Stackexchange messed up the code after adding another image. $\endgroup$ – pink vertex Nov 29 '14 at 17:21
4
$\begingroup$

An alternative way to @ideasman42's answer is to first create a 3x4 P matrix (as used in computer vision) from blender's camera parameters and use that to project. This might be convenient for you since you seem to be doing computer vision work.

Use the function get_3x4_RT_matrix_from_blender listed in my answer to 3x4 camera matrix from blender camera to generate a P matrix, then project. For instance:

# Insert your camera name here
cam = bpy.data.objects['Camera.001']
P = get_3x4_P_matrix_from_blender(cam)
e1 = Vector((1, 0,    0, 1))
p1 = P * e1
p1 /= p1[2]
print("Projected e1")
print(p1)

For vertices in the object's local coordinates, you will have to first transform them to world coordinates. Here's a sketch of how to do it:

location, rotation, scale = obj.matrix_world.decompose()
R = rotation.to_matrix().transposed();
p = R*scale*vertex_local_coordinates + location
# now apend coordinate 1 to p, use P*p then divide by 3rd coordinate

You can compare that to @ideasman42's code which I made into a function project_by_object_utils(cam, point) (see my answer to 3x4 camera matrix from blender camera). Both routines return exactly the same coordinates up to many decimal places.

  • Note: The origin is at the top-left corner and image y axis pointing down.
$\endgroup$
0
$\begingroup$

This is my code:

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

#renderer an image and save to F drive
scene.render.image_settings.file_format = 'PNG'
scene.render.filepath = "F:/image.png"
bpy.ops.render.render(write_still = 1)

width =  scene.render.resolution_x 
height = scene.render.resolution_y

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.1

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get a bvh tree and vertices
mWorld = obj.matrix_world
vertices = [mWorld @ v.co for v in obj.data.vertices]

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    cubeMesh=bpy.ops.mesh.primitive_cube_add(location=(v))
    bpy.ops.transform.resize(value=(0.01, 0.01, 0.01))
    cubeObject=bpy.context.object

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0 and co2D.z >0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        location= scene.ray_cast(bpy.context.view_layer, cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if location[0] and (v - location[1]).length < limit:
            print("%.2f %.2f" %(co2D.x*width,co2D.y*height))
    bpy.data.objects.remove(cubeObject,do_unlink= True)
$\endgroup$
  • $\begingroup$ What does your code do? Can you explain what this achieves and how? $\endgroup$ – Duarte Farrajota Ramos Oct 22 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.