I am looking to annotate points on a rendered 2D image and then use the rendered depth map with the camera intrinsics to find where the point in 3D space is. This process is identical to creating a point cloud from a depth map instead I am looking for select points. I am following the function from the answer supplied here.
def point_cloud(self, depth): """Transform a depth image into a point cloud with one point for each pixel in the image, using the camera transform for a camera centred at cx, cy with field of view fx, fy. depth is a 2-D ndarray with shape (rows, cols) containing depths from 1 to 254 inclusive. The result is a 3-D array with shape (rows, cols, 3). Pixels with invalid depth in the input have NaN for the z-coordinate in the result. """ rows, cols = depth.shape c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True) valid = (depth > 0) & (depth < 255) z = np.where(valid, depth / 256.0, np.nan) x = np.where(valid, z * (c - self.cx) / self.fx, 0) y = np.where(valid, z * (r - self.cy) / self.fy, 0) return np.dstack((x, y, z))
From reading about pinhole camera's from here I learned that:
In a true pinhole camera, both fx and fy have the same value
So i have set
self.fy to 50
I have entered the cameras position in blender which is (0,1,0) so:
self.cx = 0
self.cy = 1
Renders and Compositor:
import cv2 import numpy as np def point_cloud(depth): rows, cols = depth.shape c, r = np.meshgrid(np.arange(cols), np.arange(rows), sparse=True) valid = (depth > 0) & (depth < 255) z = np.where(valid, depth / 256.0, np.nan) x = np.where(valid, z * (c - 0) / .05, 0) y = np.where(valid, z * (r - 1) / .05, 0) return np.dstack((x, y, z)) img = cv2.imread('/home/sam/Desktop/Depth0001.png', cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH) img = img[:,:,1] points = point_cloud(img)
This method does not output the correct set of points, I would appreciate if anyone can show me what I am doing wrong.