# Get pixel coord in rendered image of pixel coord in texture

## Problem:

As shown in the figure above, I have the coordinate pair (x, y) which describes a certain point in an image texture and I want the corresponding coordinate pair (x_, y_) which describes the same point in the texture inside of the rendered image.

## Details:

The texture is applied onto a plane primitive and the texture mapping mode is "Generated", not "UV". The plane as well as the camera can be rotated and translated arbitrarily, but not to such an extent that the front side of the plane is not visible in the rendering anymore. Furthermore, the plane is scaled in x-direction to match the texture's aspect ratio. I am using Blender 2.79.

1. Calculate the corresponding (x, y, z) coordinate triple of the point p on the plane's surface in 3D-space, onto which the point of interest gets mapped in object space.

2. Project this 3D-point p in object space into the 2D-space of the rendered image via world space and camera space.

The plane's vertices are:

(-1.0000, -1.0000, 0.0000)
(1.0000, -1.0000, 0.0000)
(-1.0000, 1.0000, 0.0000)
(1.0000, 1.0000, 0.0000)


as confirmed by:

for v in sign.data.vertices: print(v.co)


while texture coordinates lie within [0, tex_width] for x and [0, tex_height] for y.

Therefore, p.x, p.y and p.z should be:

p.x = x / tex_width * 2 - 1
p.y = y / tex_height * 2 - 1
p.z = 0


(Divide by texture width and height to rescale coordinates to range [0, 1], multiply by plane width and height (both equal to 2 blender units) to rescale coordinates to range [0, 2] and subtract 1 to shift to range [-1, 1].)

The point p should be projected onto the image plane via the model-view-projection matrix (MVP-matrix) which is composed as follows:

mvp_matrix = projection_matrix * view_matrix * model_matrix


All four matrices are 4x4. For projection, p is given a fourth component w = 1, to make it homogenous. Now it can be projected to p_ in camera space as follows:

p_ = mvp_matrix * p


The coordinates of p_ should be in range [0, 1]. The desired coordinates (x_, y_) can then be calculated by rescaling p_.x and p_.y to the ranges [0, res_x], [0, res_y], res_x and res_y being the resolution of the rendered image:

x_ = p_.x * res_x
y_ = p_.y * res_y


## Code:

import math
from mathutils import Vector
import bpy

# set up scene
scn = bpy.context.scene
scn.render.engine = "CYCLES"
for o in bpy.data.objects:
if o.type == "MESH":  # delete cube

# load texture, calculate aspect ratio
tex_w, tex_h = im_tex.size
ratio = tex_w/tex_h

# add plane and rescale, rotate and translate it
plane = bpy.context.active_object
plane.scale = (ratio, 1., 1.)
plane.location = (2, 10, 1)

# set up material and apply texture
mat = bpy.data.materials.new("Material")
mat.use_nodes = True
mat.node_tree.nodes.clear()
plane.data.materials.append(mat)

# set up camera, rotate it and translate it
cam = bpy.context.scene.camera
cam.location = (0, -1, 0)

# render and save blend-file for debugging
bpy.ops.wm.save_as_mainfile(filepath="debug.blend")
scn.render.filepath = "rendering.png"
bpy.ops.render.render(write_still=True, use_viewport=True)

# now for the interesting part:

(x, y) = (150, 90)  # arbitrary point on texture
# Let's assume the texture resolution is (745, 543)

p = Vector((x * 2 / tex_w - 1.0,
y * 2 / tex_h - 1.0,
0,
1))

res_x = scn.render.resolution_x * scn.render.resolution_percentage / 100
res_y = scn.render.resolution_y * scn.render.resolution_percentage / 100
asp_x = scn.render.pixel_aspect_x
asp_y = scn.render.pixel_aspect_y

# Projection matrix:
p_mtx = cam.calc_matrix_camera(res_x, res_y, asp_x, asp_y)
# This corresponds to the model-view-matrix:
mv_mtx = cam.matrix_world.inverted() * plane.matrix_world.copy()
mvp_mtx = p_mtx * mv_mtx

p_ = mvp_mtx * p

p_x, p_y, _, _ = p_
x_ = res_x * p_x
y_ = res_y * p_y

print(x_, y_)


The result for this example is 4471.2872314453125 851.4681959152222, which isn't even in the rendering which has a resolution of (960, 540). The desired output would be something like 692.0, 104.5).

## Questions:

• Is there any error in my subtask theories?
• What is wrong with my implementation?

## Appendix:

texture.png:

Resolution (745, 543), mark at (150, 90)

rendering.png:

• The mark should be considered from the bottom left, I think (but this is probably not the only issue). – lemon Sep 28 '19 at 7:42
• Have a look to world_to_camera_view docs.blender.org/api/current/… (ok for 2.79 too) and eventually its source code: github.com/sobotka/blender/blob/… – lemon Sep 28 '19 at 7:44
• @lemon: Thanks, these were great hints! – S818 Sep 28 '19 at 18:30

Let's ignore the aspect ratio for now. The projection matrix function may sound cool, but I would settle for the easier world_to_camera_view which lemon suggests in the comments.

Y-coordinates increase upwards (bottom = 0, top = 1).

1. Get the world coordinate of the point by applying the object matrix.
2. Get the relative camera coordinate using world_to_camera_view.
3. Get the absolute render coordinate by multiplying with the resolution.

Note, that we require a scene.update() call before using the plane.matrix_world.

(x, y) = (150, 543-90)  # arbitrary point on texture
# Let's assume the texture resolution is (748, 543)

res_x = scn.render.resolution_x * scn.render.resolution_percentage / 100
res_y = scn.render.resolution_y * scn.render.resolution_percentage / 100

p = Vector((x * 2 / tex_w - 1.0, y * 2 / tex_h - 1.0, 0))

bpy.context.scene.update()
p_camera = world_to_camera_view(bpy.context.scene, cam, plane.matrix_world * p)

p_camera.x *= res_x
p_camera.y *= res_y
print(p_camera)

• By the way, if you prepare code for others: Nobody can understand pv_mtx, x_myt or pmt_xxty without comments. – Leander Sep 28 '19 at 10:29
• Thank you, this works great! I changed only one thing: p_camera.y = 540 - (p_camera.y * res_y) This way the y-coordinate is counted from the top again, my preferred convention for the rest of my script. As for your comment: I tried to choose good variable names which are not too long to keep the code short and readable, but I'll try harder next time and choose more explicit names! – S818 Sep 28 '19 at 18:35
• Just one small follow up question: What is the scene update supposed to do? The code also seems to work without it. Maybe it can go wrong without the update, but doesn't necessarily? – S818 Sep 28 '19 at 18:58
• Ah, nevermind. I think I found the answer: https://docs.blender.org/api/2.79/info_gotcha.html#no-updates-after-setting-values – S818 Sep 28 '19 at 19:05
• @S818 A short variable name with uncertain meaning has much worse readability than a long name with unambiguous meaning. The length would have to be ridiculous before the reverse holds true. Even say, the_camera_coordinate_lying_on_the_y_axis, as silly as that is, is better than, for example, c_y which is not readable at all. (This is for shared code. For private code do whatever you want...though best to practice good habits there, too.) – B Layer Feb 26 '20 at 6:43