How to set object at a specific depth from the camera, at a certain pixel

Question

I want to set an object at a certain distance (or depth) relative to the camera; and also at a particular pixel.

Example:

I set up a camera at a random position in blender. I want an 1280x720 image. So if a line came out of the camera at pixel e.g. (400,400), perpendicular to the image plane; and that line travels 1300 millimeters. It will end up at position (x,y,z) in the blender world. That is the point where I want the bottom centre of my object to be at. How do I calculate this location (x,y,z) ?

---

Some more context here. I have a image of a scene with a table. I also have the corresponding depth image, which stores for each pixel, the distance in millimeter from the scene to the camera. I have the segmentation mask of this table. So I know for each point on the table, its distance to the camera. I want to set an object on this table, by rendering the scene as background, and rendering the object on top of it at the correct location from the camera.

Answer 1

I think, next must be works:

Sets empty objects to every corner of image - pinLT, pinRT, pin_BL, pinBR. Then calculate rays from camera base to those pins:

eye = camera.matrix_world @ Vector((0,0,0))
rayLT =  (pinLT.location @ pinLT.matrix_world) - eye
rayRT =  (pinRT.location @ pinRT.matrix_world) - eye
rayLB =  (pinLT.location @ pinLB.matrix_world) - eye
#rayRB =  (pinRT.location @ pinRB.matrix_world) - eye

then make interpolation to find correspond ray to the pixel

h = (rayRT - rayLT) * pixels.x / pixels.wide
v = (rayLB - rayLT) * pixels.y / pixels.height

ray = pinLT + v + h

# calc result in world coorinates
resultOnImage = eye + ray
resultInWorld = resultOnImage + ray.normalized() * desired_distance

Answer 2

This is called the Inverse Projection Transformation or back-projection: 2D pixel -> 3D world coordinate. I only needed the 3D world coordinate relative to the camera. This worked for me:

1. Set up the camera at location (0, 0, 0) and rotation (90, 0, 0) (degrees) with the correct focal length and sensor size.

2. Get the camera intrinsics Fx, Fy (focal lengths for x and y) and Cx, Cy (optical centre)

3. Do the back-projection. Here (Px, Py) is your pixel-coordinate; Z is the depth at that pixel-coordinate (received from the depth map I have).

x = (px - cx) * z / fx

y = (py - cy) * z / fy

Your world coordinate is then (x, y, z) relative to the camera. So with my camera setup, I can set the object at location (x, z, -y) (divided by 1000, because I used millimeter as measurement for the depth).