# Calculating 3D world co-ordinates using depth map and camera intrinsics

I am looking to annotate points on a rendered 2D image and then use the rendered depth map with the camera intrinsics to find where the point in 3D space is. This process is identical to creating a point cloud from a depth map instead I am looking for select points. I am following the function from the answer supplied here.

Function:

def point_cloud(self, depth):
"""Transform a depth image into a point cloud with one point for each
pixel in the image, using the camera transform for a camera
centred at cx, cy with field of view fx, fy.

depth is a 2-D ndarray with shape (rows, cols) containing
depths from 1 to 254 inclusive. The result is a 3-D array with
shape (rows, cols, 3). Pixels with invalid depth in the input have
NaN for the z-coordinate in the result.

"""
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
valid = (depth > 0) & (depth < 255)
z = np.where(valid, depth / 256.0, np.nan)
x = np.where(valid, z * (c - self.cx) / self.fx, 0)
y = np.where(valid, z * (r - self.cy) / self.fy, 0)
return np.dstack((x, y, z))


self.fx & self.fy:

$$\hspace{.5cm}$$

From reading about pinhole camera's from here I learned that:

In a true pinhole camera, both fx and fy have the same value

So i have set self.fx and self.fy to 50

self.cx & self.cy:

I have entered the cameras position in blender which is (0,1,0) so: self.cx = 0 self.cy = 1

Renders and Compositor:

$$\hspace{5cm}$$

$$\hspace{5cm}$$

My Implementation:

import cv2
import numpy as np

def point_cloud(depth):
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
valid = (depth > 0) & (depth < 255)
z = np.where(valid, depth / 256.0, np.nan)
x = np.where(valid, z * (c - 0) / .05, 0)
y = np.where(valid, z * (r - 1) / .05, 0)
return np.dstack((x, y, z))

img = img[:,:,1]
points = point_cloud(img)


This method does not output the correct set of points, I would appreciate if anyone can show me what I am doing wrong.

• PNG is not a good choice to store depth data as is a display referred format. If you must use such a format make sure at least that you are not encoding color transforms set in the color management to keep the data linear. For this kind of purposes consider using EXR for all the data, as you can store depth information linearly and you are not constrained by the bit depth of the image, no need to normalize or scale the data.
– susu
Dec 11, 2020 at 19:58
• I chose PNG as this code deals with a pixel depth of 8 bits, I plan to use to the full floating point accuracy in the exr format but for now I just want to get this method working.
– Sam
Dec 12, 2020 at 15:00

Several things to take into account to rebuild the rendered mesh.

• Camera parameters: Focal angle, in particular to reconstitute the mesh with the good positions. See here.

Also, camera clipping planes can be used to filter the image.

• Camera axis: Camera's Z is to the opposite to the projection: we need to minus Z from the image.

The rendered image is like a mirror: we need to minus X from the image.

• Image size: If camera is not shifted (common case), it is at the center of the image.

So coordinates should be centered around the half of the image width and height and scaled in consequence.

• Compositor node: Don't normalize the depth. If normalized, the values have no mean: they are just remapped between 0 and 1.

• Image format: Don't use PNG. You can't store the wanted values in a PNG except if normalizing them and we don't want that.

Use OpenEXR which allows to store the Z values as they are.

Compositor settings

Use OpenEXR and don't normalize:

Code

Comments below for key elements:

import bpy
import cv2
import numpy as np
from math import tan
from mathutils import Vector

def point_cloud(depth,cam):

# Distance factor from the cameral focal angle
factor = 2.0 * tan(cam.data.angle_x/2.0)

rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
# Valid depths are defined by the camera clipping planes
valid = (depth > cam.data.clip_start) & (depth < cam.data.clip_end)

# Negate Z (the camera Z is at the opposite)
z = -np.where(valid, depth, np.nan)
# Mirror X
# Center c and r relatively to the image size cols and rows
ratio = max(rows,cols)
x = -np.where(valid, factor * z * (c - (cols / 2)) / ratio, 0)
y = np.where(valid, factor * z * (r - (rows / 2)) / ratio, 0)

return np.dstack((x, y, z))

img = img[:,:,1]

# Get the camera
cam = bpy.data.objects['Camera']

# Calculate the points
points = point_cloud(img, cam)

# Get the camera matrix for location conversion
cam_mat = cam.matrix_world

# Translate the points
verts = [cam_mat @ Vector(p) for r in points for p in r]

# Create a mesh from the points
mesh_data = bpy.data.meshes.new("result")
mesh_data.from_pydata(verts, [], [])
mesh_data.update()

# Create an object with this mesh
obj = bpy.data.objects.new("result", mesh_data)

# Link it to the scene
scene = bpy.context.scene