Obtain normal map from specific camera and transformation

Question

This may be a long shot, but after researching fruitlessly for about two weeks, I wanted to ask here in the hopes that someone who is a bit more knowledgeable about Blender and computer vision can assist me or at least point me in the right direction. The problem is, I would like to know how I can acquire the normal map specific to a camera and then transform it based on another camera.

The scenario is, let's say we have stereo cameras; Camera A and Camera B. We would then want to obtain the normal map A' that comes from transforming the normal map of a view point A (Camera A) to another view point B (Camera B). Then perform matching in two normal maps: A' and B, similar to SURF or SIFT for finding the correspondences between these two normal maps.

The limitations I have encountered are: 1) How to acquire the normal map (camera specific) without baking? 2) How to transform that normal map (once acquired) to Camera B?

I assume using opencv to find the correspondences between the two normal maps would be the easiest part, so these two questions are most important for me at the moment. Also tips on how to apply this in cycles/Sverchok would also be very much appreciated.

Answer 1

In Cycles, the Normal pass can be enabled in the render by enabling the Normal pass in the Render Layer properties - however, this is in World coordinates, not Camera coordinates. To convert the Normal into Camera space we can use the Vector Transform node. This isn't possible with the actual Normal render pass but we can add an additional Render Layer and configure that to generate the "camera-space normal" render.

In order to do this we first need to create a material which will generate the normal in camera space. This can be achieved with the following nodes :

This takes the geometry Normal (in World space) and transforms it into Camera space via the Vector Transform node (notice how it's set to 'Normal' mode, with World to Camera). This then drives the Emission Shader node to generate the output.

In order to generate this as an additional pass for the whole scene we need to add a Render Layer and set its Material Override to the new material shown above - so that all objects are rendered with this material.

The output of that Render Layer will now be available in the compositor and represents the Normal In Camera Space.

This can then be repeated for both cameras (you can actually use Render Layer 'Views' set to Multi-view to render both cameras automatically).

In order to transform the result into the opposite camera space we need to apply a transform to the generated normal to rotate by the difference in angle between the cameras. For this I've made an assumption that you already know the angular difference between the cameras and that the rotation is purely in the horizontal direction of both cameras - ie, in the same way our eyes in the same horizontal plane.

For this we can simply rotate by the specified angle around the Y axis using some vector maths :

Here we're transforming the R element (x-axis) to cos(angle)*R+sin(angle)*B and the B element (z-axis) to -sin(angle)*R+cos(angle)*B - effectively a rotation matrix - and leaving G (y-axis) unchanged. Set the Value node to the angle of rotation (the angle between your cameras) - in Radians.

This can produce the following result (slightly different to the unrotated result) :

Blend file included