# How to project back rendered depth maps to recover the ground-truth 3D shape using Blender?

You can discard the images I have uploaded and can simply use some camera angles of your choice to render depth maps of a 3D shape and then fuse the rendered depth maps to recover the underlying 3D shape

I recently published a paper on 3D shape generation in a computer vision conference (CVPR). My co-author wrote the code (in C++ and using OpenCV) for fusing the depth maps and getting the final 3D shapes from the produced multi-view outputs. The inputs to the code he wrote is 20 depth maps, ground-truth camera angles (posted below) and the distance from the centroid of the shapes to the camera (distance to the shape centroid=1.5 on a sphere). The centroid of shapes are calculated as follow:

First, the centers of faces (triangle) of a mesh centroid of a mesh is calculated. Then the faces areas are computed. The new centroid is the average of the mesh faces' centers, weighted by their area.

Here some people have written algorithms on how to compute the centroids.

Unfortunately my friend is not available to help me on this and I cannot use OpenCV and C++ for this new project that I'm beginning to work on. So any help would be appreciated. My goal is to write some code using Blender's Python API instead of using my co-author's C++ code to do the same thing. But before I move on, I wonder if Blender has some built-in functions that can generate the final 3D shape given rendered depth maps of that shape, camera angles and the distance to the camera? If not, can anyone give me some ideas on how I should do that and give me a code sample for it?

Here I have uploaded a set of rendered depth maps a headphone's 3D shape that you can use for backward projection (reconstructing the 3D shape). And if you prefer to start with a 3D shape directly, here you can download a 3D shape of a different headphone we used in our work before. You can render depth maps the 3D shape using the camera angles posted below.

FYI, here is my co-author's high-level description on how his approach works:

In the final step, all depth maps are projected back to the 3D space to create the final rendering. We reconstruct 3D shapes from multi-view silhouettes and depth maps by first generating a 3D point cloud from each depth image with its corresponding camera setting (x, y, z coordinates). The union of these point clouds from all views can be seen as an initial estimation of the shape.

And here are the camera angles we used for doing the rendering in the first place:

-0.57735  -0.57735  0.57735
0.934172  0.356822  0
0.934172  -0.356822  0
-0.934172  0.356822  0
-0.934172  -0.356822  0
0  0.934172  0.356822
0  0.934172  -0.356822
0.356822  0  -0.934172
-0.356822  0  -0.934172
0  -0.934172  -0.356822
0  -0.934172  0.356822
0.356822  0  0.934172
-0.356822  0  0.934172
0.57735  0.57735  -0.57735
0.57735  0.57735  0.57735
-0.57735  0.57735  -0.57735
-0.57735  0.57735  0.57735
0.57735  -0.57735  -0.57735
0.57735  -0.57735  0.57735
-0.57735  -0.57735  -0.57735

• That second link doesn't go anywhere. Preferably add the image(s) into the post directly. See: meta.stackexchange.com/questions/75491/… – Ray Mairlot Feb 7 '18 at 16:57
• @RayMairlot Sorry, I just fixed it. There are 20 depth maps so I did not want to attach them one by one here. Also because the compression methods are applied on uploaded images and you cannot use them in raw formats I have (probably) – Amir Feb 7 '18 at 18:13
• @Amir are these the angles in correspondence to the sequence of the same images? – Rick Riggs Feb 7 '18 at 21:51
• Yes Rick. I also uploaded an .obj file we used for rendering. The link is on the bottom of my post. – Amir Feb 7 '18 at 21:55
• @Amir I think your second image must be rotated 180° on its own normal. Just positioning the first two images is showing this conflict. Is the original code doing this to the output? Also do you happen to know camera distance from the subject? – Rick Riggs Feb 7 '18 at 22:34

After looking at the following image from your paper, it seems to me as if your image is basically a sprite map, so I propose the following workflow to get you there.

Proposed Workflow:

I'm certain that you could mimic the polyhedron shape in blender to set active planes (viewports) on programatically, and use an UV coordinate mapping per sprite location to get a depth map per the corresponding camera location/angle (position).

Set up a highly subdivided plane for each view, then attach the Displacement Modifier to it, along with a custom UV layout to the sprite sheet that corresponds with the appropriate viewpoint.

I would then add a global driver to attempt to control the displacement modifier strength per projection, and when they get close to each other...

Apply the modifier, Join the objects together, and remove duplicate verts.

• Since your update... I may be mistaken as to the format of sprite image versus individual images, so in that case, just create a material per texture, and UV unwrap to each one, everything else would be the same approach. – Rick Riggs Feb 7 '18 at 18:19
• Sorry this is a bit too high-level for me since I am not an expert in Blender. But I noticed that you are talking about rendering. However, I am talking about projecting back the already rendered views to get the underlying 3D Shape. Do you have a solution for that? – Amir Feb 7 '18 at 18:34
• Using the code my co-author has written, I can use the uploaded depth maps and get the underlying 3D Shapes. – Amir Feb 7 '18 at 18:35
• @Amir if your last comment is meant a question, then yes you can. I will try and give an example of the answer I have provided once I am able to put some time into it. – Rick Riggs Feb 7 '18 at 18:38
• No that was not a question. I basically mean I feel you did not get the question I have asked. The goal is not to render, is to produce 3D. – Amir Feb 7 '18 at 18:45