Calculating 3D world co-ordinates using depth map and camera intrinsics

Question

I am looking to annotate points on a rendered 2D image and then use the rendered depth map with the camera intrinsics to find where the point in 3D space is. This process is identical to creating a point cloud from a depth map instead I am looking for select points. I am following the function from the answer supplied here.

Function:

def point_cloud(self, depth):
"""Transform a depth image into a point cloud with one point for each
pixel in the image, using the camera transform for a camera
centred at cx, cy with field of view fx, fy.

depth is a 2-D ndarray with shape (rows, cols) containing
depths from 1 to 254 inclusive. The result is a 3-D array with
shape (rows, cols, 3). Pixels with invalid depth in the input have
NaN for the z-coordinate in the result.

"""
rows, cols = depth.shape
c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
valid = (depth > 0) & (depth < 255)
z = np.where(valid, depth / 256.0, np.nan)
x = np.where(valid, z * (c - self.cx) / self.fx, 0)
y = np.where(valid, z * (r - self.cy) / self.fy, 0)
return np.dstack((x, y, z))

self.fx & self.fy:

$\hspace{.5cm}$

From reading about pinhole camera's from here I learned that:

In a true pinhole camera, both fx and fy have the same value

So i have set self.fx and self.fy to 50

self.cx & self.cy:

I have entered the cameras position in blender which is (0,1,0) so: self.cx = 0 self.cy = 1

Renders and Compositor:

$\hspace{5cm}$

$\hspace{5cm}$

My Implementation:

import cv2
import numpy as np

def point_cloud(depth):
    rows, cols = depth.shape
    c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
    valid = (depth > 0) & (depth < 255)
    z = np.where(valid, depth / 256.0, np.nan)
    x = np.where(valid, z * (c - 0) / .05, 0)
    y = np.where(valid, z * (r - 1) / .05, 0)
    return np.dstack((x, y, z))

img = cv2.imread('/home/sam/Desktop/Depth0001.png',  cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
img = img[:,:,1]
points = point_cloud(img)

This method does not output the correct set of points, I would appreciate if anyone can show me what I am doing wrong.

Answer 1

Several things to take into account to rebuild the rendered mesh.

• Camera parameters: Focal angle, in particular to reconstitute the mesh with the good positions. See here.

  Also, camera clipping planes can be used to filter the image.

• Camera axis: Camera's Z is to the opposite to the projection: we need to minus Z from the image.

  The rendered image is like a mirror: we need to minus X from the image.

• Image size: If camera is not shifted (common case), it is at the center of the image.

  So coordinates should be centered around the half of the image width and height and scaled in consequence.

• Compositor node: Don't normalize the depth. If normalized, the values have no mean: they are just remapped between 0 and 1.

• Image format: Don't use PNG. You can't store the wanted values in a PNG except if normalizing them and we don't want that.

  Use OpenEXR which allows to store the Z values as they are.

Compositor settings

Use OpenEXR and don't normalize:

Code

Comments below for key elements:

import bpy
import cv2
import numpy as np
from math import tan
from mathutils import Vector

def point_cloud(depth,cam):
    
    # Distance factor from the cameral focal angle
    factor = 2.0 * tan(cam.data.angle_x/2.0)
    
    rows, cols = depth.shape
    c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True)
    # Valid depths are defined by the camera clipping planes
    valid = (depth > cam.data.clip_start) & (depth < cam.data.clip_end)
    
    # Negate Z (the camera Z is at the opposite)
    z = -np.where(valid, depth, np.nan)
    # Mirror X
    # Center c and r relatively to the image size cols and rows
    ratio = max(rows,cols)
    x = -np.where(valid, factor * z * (c - (cols / 2)) / ratio, 0)
    y = np.where(valid, factor * z * (r - (rows / 2)) / ratio, 0)
    
    return np.dstack((x, y, z))

img = cv2.imread('C:/tmp/depth0001.exr',  cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
img = img[:,:,1]

# Get the camera
cam = bpy.data.objects['Camera']

# Calculate the points
points = point_cloud(img, cam)

# Get the camera matrix for location conversion
cam_mat = cam.matrix_world

# Translate the points
verts = [cam_mat @ Vector(p) for r in points for p in r]

# Create a mesh from the points
mesh_data = bpy.data.meshes.new("result")
mesh_data.from_pydata(verts, [], [])
mesh_data.update()

# Create an object with this mesh
obj = bpy.data.objects.new("result", mesh_data)

# Link it to the scene
scene = bpy.context.scene
scene.collection.objects.link(obj)

Results

Test file (relocate the compositor output before using it, and the file location in the code)

Also note (if useful), it seems that color management has no influence for depth.