4
$\begingroup$

enter image description here

Problem:

As shown in the figure above, I have the coordinate pair (x, y) which describes a certain point in an image texture and I want the corresponding coordinate pair (x_, y_) which describes the same point in the texture inside of the rendered image.

Details:

The texture is applied onto a plane primitive and the texture mapping mode is "Generated", not "UV". The plane as well as the camera can be rotated and translated arbitrarily, but not to such an extent that the front side of the plane is not visible in the rendering anymore. Furthermore, the plane is scaled in x-direction to match the texture's aspect ratio. I am using Blender 2.79.


The task can be split into to subtasks:

  1. Calculate the corresponding (x, y, z) coordinate triple of the point p on the plane's surface in 3D-space, onto which the point of interest gets mapped in object space.

  2. Project this 3D-point p in object space into the 2D-space of the rendered image via world space and camera space.

Theory for subtask 1:

The plane's vertices are:

(-1.0000, -1.0000, 0.0000)
(1.0000, -1.0000, 0.0000)
(-1.0000, 1.0000, 0.0000)
(1.0000, 1.0000, 0.0000)

as confirmed by:

for v in sign.data.vertices: print(v.co)

while texture coordinates lie within [0, tex_width] for x and [0, tex_height] for y.

Therefore, p.x, p.y and p.z should be:

p.x = x / tex_width * 2 - 1
p.y = y / tex_height * 2 - 1
p.z = 0

(Divide by texture width and height to rescale coordinates to range [0, 1], multiply by plane width and height (both equal to 2 blender units) to rescale coordinates to range [0, 2] and subtract 1 to shift to range [-1, 1].)

Theory for subtask 2:

The point p should be projected onto the image plane via the model-view-projection matrix (MVP-matrix) which is composed as follows:

mvp_matrix = projection_matrix * view_matrix * model_matrix

All four matrices are 4x4. For projection, p is given a fourth component w = 1, to make it homogenous. Now it can be projected to p_ in camera space as follows:

p_ = mvp_matrix * p

The coordinates of p_ should be in range [0, 1]. The desired coordinates (x_, y_) can then be calculated by rescaling p_.x and p_.y to the ranges [0, res_x], [0, res_y], res_x and res_y being the resolution of the rendered image:

x_ = p_.x * res_x
y_ = p_.y * res_y

Code:

import math
from mathutils import Vector
import bpy

# set up scene
scn = bpy.context.scene
scn.render.engine = "CYCLES"
for o in bpy.data.objects:
    if o.type == "MESH":  # delete cube
        bpy.data.objects.remove(o, do_unlink=True)

# load texture, calculate aspect ratio
im_tex = bpy.data.images.load("texture.png")
tex_w, tex_h = im_tex.size
ratio = tex_w/tex_h

# add plane and rescale, rotate and translate it
bpy.ops.mesh.primitive_plane_add()
plane = bpy.context.active_object
plane.scale = (ratio, 1., 1.)
plane.location = (2, 10, 1)
plane.rotation_euler = (math.radians(90),
                        math.radians(5),
                        math.radians(-15))

# set up material and apply texture
mat = bpy.data.materials.new("Material")
mat.use_nodes = True
mat.node_tree.nodes.clear()
input_node = mat.node_tree.nodes.new(type="ShaderNodeTexCoord")
tex_node = mat.node_tree.nodes.new(type="ShaderNodeTexImage")
tex_node.image = bpy.data.images.load("texture.png")
bsdf_node = mat.node_tree.nodes.new(type='ShaderNodeBsdfPrincipled')
output_node = mat.node_tree.nodes.new(type='ShaderNodeOutputMaterial')
links = mat.node_tree.links
link_0 = links.new(input_node.outputs["Generated"], tex_node.inputs["Vector"])
link_1 = links.new(tex_node.outputs["Color"], bsdf_node.inputs["Base Color"])
link_2 = links.new(bsdf_node.outputs["BSDF"], output_node.inputs["Surface"])
plane.data.materials.append(mat)

# set up camera, rotate it and translate it
cam = bpy.context.scene.camera
cam.rotation_euler = (math.radians(90), 0, math.radians(5))
cam.location = (0, -1, 0)

# render and save blend-file for debugging
bpy.ops.wm.save_as_mainfile(filepath="debug.blend")
scn.render.filepath = "rendering.png"
bpy.ops.render.render(write_still=True, use_viewport=True)

# now for the interesting part:

(x, y) = (150, 90)  # arbitrary point on texture
                     # Let's assume the texture resolution is (745, 543)

# subtask 1:
p = Vector((x * 2 / tex_w - 1.0,
            y * 2 / tex_h - 1.0,
            0,
            1))

res_x = scn.render.resolution_x * scn.render.resolution_percentage / 100
res_y = scn.render.resolution_y * scn.render.resolution_percentage / 100
asp_x = scn.render.pixel_aspect_x
asp_y = scn.render.pixel_aspect_y

# Projection matrix:
p_mtx = cam.calc_matrix_camera(res_x, res_y, asp_x, asp_y)
# This corresponds to the model-view-matrix:
mv_mtx = cam.matrix_world.inverted() * plane.matrix_world.copy()
mvp_mtx = p_mtx * mv_mtx

# subtask 2:
p_ = mvp_mtx * p

p_x, p_y, _, _ = p_
x_ = res_x * p_x
y_ = res_y * p_y

print(x_, y_)

The result for this example is 4471.2872314453125 851.4681959152222, which isn't even in the rendering which has a resolution of (960, 540). The desired output would be something like 692.0, 104.5).

Questions:

  • Is there any error in my subtask theories?
  • What is wrong with my implementation?

Appendix:

texture.png:

Resolution (745, 543), mark at (150, 90) enter image description here

rendering.png:

enter image description here

$\endgroup$
  • 1
    $\begingroup$ The mark should be considered from the bottom left, I think (but this is probably not the only issue). $\endgroup$ – lemon Sep 28 at 7:42
  • 1
    $\begingroup$ Have a look to world_to_camera_view docs.blender.org/api/current/… (ok for 2.79 too) and eventually its source code: github.com/sobotka/blender/blob/… $\endgroup$ – lemon Sep 28 at 7:44
  • $\begingroup$ @lemon: Thanks, these were great hints! $\endgroup$ – S818 Sep 28 at 18:30
3
$\begingroup$

Let's ignore the aspect ratio for now. The projection matrix function may sound cool, but I would settle for the easier world_to_camera_view which lemon suggests in the comments.

Y-coordinates increase upwards (bottom = 0, top = 1).

  1. Get the world coordinate of the point by applying the object matrix.
  2. Get the relative camera coordinate using world_to_camera_view.
  3. Get the absolute render coordinate by multiplying with the resolution.

Note, that we require a scene.update() call before using the plane.matrix_world.

(x, y) = (150, 543-90)  # arbitrary point on texture
                        # Let's assume the texture resolution is (748, 543)

res_x = scn.render.resolution_x * scn.render.resolution_percentage / 100
res_y = scn.render.resolution_y * scn.render.resolution_percentage / 100

p = Vector((x * 2 / tex_w - 1.0, y * 2 / tex_h - 1.0, 0))

bpy.context.scene.update()
p_camera = world_to_camera_view(bpy.context.scene, cam, plane.matrix_world * p)

p_camera.x *= res_x
p_camera.y *= res_y
print(p_camera)
$\endgroup$
  • 1
    $\begingroup$ By the way, if you prepare code for others: Nobody can understand pv_mtx, x_myt or pmt_xxty without comments. $\endgroup$ – Leander Sep 28 at 10:29
  • $\begingroup$ Thank you, this works great! I changed only one thing: p_camera.y = 540 - (p_camera.y * res_y) This way the y-coordinate is counted from the top again, my preferred convention for the rest of my script. As for your comment: I tried to choose good variable names which are not too long to keep the code short and readable, but I'll try harder next time and choose more explicit names! $\endgroup$ – S818 Sep 28 at 18:35
  • $\begingroup$ Just one small follow up question: What is the scene update supposed to do? The code also seems to work without it. Maybe it can go wrong without the update, but doesn't necessarily? $\endgroup$ – S818 Sep 28 at 18:58
  • $\begingroup$ Ah, nevermind. I think I found the answer: https://docs.blender.org/api/2.79/info_gotcha.html#no-updates-after-setting-values $\endgroup$ – S818 Sep 28 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.