Calculating 3D world co-ordinates using depth map and camera intrinsics

I am looking to annotate points on a rendered 2D image and then use the rendered depth map with the camera intrinsics to find where the point in 3D space is. This process is identical to creating a point cloud from a depth map instead I am looking for select points. I am following the function from the answer supplied here.

Function:

def point_cloud(self, depth):
"""Transform a depth image into a point cloud with one point for each
pixel in the image, using the camera transform for a camera
centred at cx, cy with field of view fx, fy.

depth is a 2-D ndarray with shape (rows, cols) containing
depths from 1 to 254 inclusive. The result is a 3-D array with
shape (rows, cols, 3). Pixels with invalid depth in the input have
NaN for the z-coordinate in the result.

"""
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
valid = (depth > 0) & (depth < 255)
z = np.where(valid, depth / 256.0, np.nan)
x = np.where(valid, z * (c - self.cx) / self.fx, 0)
y = np.where(valid, z * (r - self.cy) / self.fy, 0)
return np.dstack((x, y, z))

self.fx & self.fy:

$$\hspace{.5cm}$$ In a true pinhole camera, both fx and fy have the same value

So i have set self.fx and self.fy to 50

self.cx & self.cy:

I have entered the cameras position in blender which is (0,1,0) so: self.cx = 0 self.cy = 1

Renders and Compositor:

$$\hspace{5cm}$$ $$\hspace{5cm}$$  My Implementation:

import cv2
import numpy as np

def point_cloud(depth):
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
valid = (depth > 0) & (depth < 255)
z = np.where(valid, depth / 256.0, np.nan)
x = np.where(valid, z * (c - 0) / .05, 0)
y = np.where(valid, z * (r - 1) / .05, 0)
return np.dstack((x, y, z))

img = img[:,:,1]
points = point_cloud(img)

This method does not output the correct set of points, I would appreciate if anyone can show me what I am doing wrong.

• PNG is not a good choice to store depth data as is a display referred format. If you must use such a format make sure at least that you are not encoding color transforms set in the color management to keep the data linear. For this kind of purposes consider using EXR for all the data, as you can store depth information linearly and you are not constrained by the bit depth of the image, no need to normalize or scale the data.
– susu
Dec 11 '20 at 19:58
• I chose PNG as this code deals with a pixel depth of 8 bits, I plan to use to the full floating point accuracy in the exr format but for now I just want to get this method working. Dec 12 '20 at 15:00

Several things to take into account to rebuild the rendered mesh.

• Camera parameters:

Focal angle, in particular to reconstitute the mesh with the good positions. See here.

Also, camera clipping planes can be used to filter the image.

• Camera axis:

Camera's Z is to the opposite to the projection: we need to minus Z from the image.

The rendered image is like a mirror: we need to minus X from the image.

• Image size:

If camera is not shifted (common case), it is at the center of the image.

So coordinates should be centered around the half of the image width and height and scaled in consequence.

• Compositor node:

Don't normalize the depth. If normalized, the values have no mean: they are just remapped between 0 and 1.

• Image format:

Don't use PNG. You can't store the wanted values in a PNG except if normalizing them and we don't want that.

Use openEXR which allows to store the Z values as they are.

Compositor settings

Use openEXR and don't normalize: Code

import bpy
import cv2
import numpy as np
from math import tan
from mathutils import Vector

def point_cloud(depth,cam):

# Distance factor from the cameral focal angle
factor = 2.0 * tan(cam.data.angle_x/2.0)

rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
# Valid depths are defined by the camera clipping planes
valid = (depth > cam.data.clip_start) & (depth < cam.data.clip_end)

# Negate Z (the camera Z is at the opposite)
z = -np.where(valid, depth, np.nan)
# Mirror X
# Center c and r relatively to the image size cols and rows
ratio = max(rows,cols)
x = -np.where(valid, factor * z * (c - (cols / 2)) / ratio, 0)
y = np.where(valid, factor * z * (r - (rows / 2)) / ratio, 0)

return np.dstack((x, y, z))

img = img[:,:,1]

# Get the camera
cam = bpy.data.objects['Camera']

# Calculate the points
points = point_cloud(img, cam)

# Get the camera matrix for location conversion
cam_mat = cam.matrix_world

# Translate the points
verts = [cam_mat @ Vector(p) for r in points for p in r]

# Create a mesh from the points
mesh_data = bpy.data.meshes.new("result")
mesh_data.from_pydata(verts, [], [])
mesh_data.update()

# Create an object with this mesh
obj = bpy.data.objects.new("result", mesh_data)

# Link it to the scene
scene = bpy.context.scene

Results Test file (relocate the compositor output before using it, and the file location in the code) Also note (if useful), it seems that color management has no influence for depth.

• hi, is it possible to reuse your script principle in Animation nodes blender.stackexchange.com/q/210536/2214 thank you for your comment Feb 12 '21 at 21:02
• @vklidu, hi. Sure, as AN can use Python scripts directly, that should be easy. (considering translating code into nodes is not a goal in itself...). Feb 14 '21 at 6:48
• I didn't mean to use script directly in AN, but use the math to recostruct depth (point cloud) by nodes ... to avoid installation of extra python modules etc ... Sure if cv2 or any other module is necessary for it's functions or speed that AN is not capable I can learn to leave with modules, since I don't have a coders skills I thought it could be easy to math the same with anim nodes. Feb 14 '21 at 7:40
• @lemon, I'm trying to adapt your code for use with numpy data structures. Specifically: cam_mat = cam.matrix_world. Where in my case, cam_mat is a numpy array and not a Mathutilis.Matrix() object. Is it possible to convert a numpy array into a Matrix object? Oct 13 '21 at 14:12
• @junfanbl, yes something like Matrix(your_numpy_array) I guess Oct 14 '21 at 10:02