8
$\begingroup$

Why Render Layer node use several outputs, like for object pass

  • CryptoObject00
  • CryptoObject02
  • CryptoObject04

and Cryptomatte node several inputs

  • Crypto00
  • Crypto01
  • Crypto02

What I tried, connecting just the first one "00" does the job. How the others are used?

I see some connection between "CryptoObject00, 02, 04, 06, ..." and "Levels" in "Passes" section of "Layer Properties" window. Levels are described as "Sets how many unique objects can be distinguished per pixel." I can imagine something in theory, but I'm not sure how these output sockets of levels? really works. Why there is not only one output with level set in properties?

Why level "2" use socket numbered "00" (level "4" = output "02" etc.)? Why it does not start "02" for level 2 if it folllows even numbering of distinguished pixels?

Similar for Cryptomatte node inputs (Crypto00, ...) - what kind of job they do and how can be color next to it useful?

enter image description here

(I saw several videos and I red wiki, forums ... but no one is explaining or asking for that. Sorry if it is so obvious, I didn't catch it.)

$\endgroup$
10
$\begingroup$

In order to explain to sockets and their numbering I'll have to explain how Cryptomatte and some aspects of path tracing work.


The Idea

Cryptomatte generates ID mattes which try to preserve for each pixel of the rendered image what elements of the 3D scene contributed to it. For instance in a render with motion blur, pixels where the moving object and background object blend into each other are influenced by both objects. In order to understand how Cryptomatte identifies and stores this information, we have to take a look at how the rendered image is created.

Rendering using a path tracer such as Cycles works roughly in the following way:

In order to create the rendered image we sample multiple paths per pixel at subpixel location from the camera into the scene. The sampling is quasi-random using either the Sobol sequences or correlated multi-jitter. For each path we check if it intersects with an object, if it does, we remember the hit point. Next we chose a random point on one of the light source and try to connect it to the hit point. This is an optimization known as Next-event prediction. If a direct path from the hit point to a light source is possible we can evaluate the shaders along the path to the camera and we receive an RGBA value. In case a connection to a light source isn't possible, we bounce with a new path segment from the hit point in a quasi-random direction and repeat the process until we either hit a light source or exceeded the maximum bounces. Cycles therefore uses Quasi-Monte-Carlo integration. In order to fuse the multiple samples per pixel in the camera plane into a final RGBA value we use a pixel filter which weights and sums the subpixel samples. Cycles uses either a Blackman-Harris, Gaussian or Box filter.

This is a very simplified and shortened explanation, for a more detailed introduction I recommend watching Lukas Stockner's talk from the Blender Conference 2019. The important part for Cryptomatte is that there are multiple samples per pixel, that may hit different objects and which therefore contribute to the final pixel.

The idea behind Cryptomatte is that we can leverage the scene and path tracing information to create ID and coverage pairs per pixel, where the ID references the element that contributed to the pixel and the coverage tells how much. This allows us later on to determine which pixel and by what amount was caused by a certain object or material.

There are three elements Cryptomatte can store ID mattes for: Objects, materials and namespaces (hierarchical groups of objects, which is the Assets option in Blender). IDs are generated by hashing the object/material/namespace's name and are stored per sample in so called arbitrary output variables (AOV).


Filtering and Encoding

Each pixel in the render is the result of the weighted sum of its samples, where each sample belongs to an ID. The weights are given by the pixel filter, such as Blackman-Harris or Gaussian filter. There can be several samples per pixel that belong to the same ID.

The weight of the samples that belong to the same ID are accumulated per pixel. Normalizing the value gives the coverage of the ID-coverage pair for that pixel. For every pixel the pairs get ranked by their coverage, rank 0 is for the highest coverage. They are then assigned to the corresponding pixel in the layer that contains said rank. Note: Each rank contain the IDs and coverages of several objects which each had the highest coverage for that specific pixel.

Multi-layer OpenEXR files are used as data structure for storing the files. Each of the AOVs is stored in a separate file. IDs and coverages are stored each in their own channel. Since OpenEXR has RGBA channels there are two ranks per layer.

Layer 1

CryptoObject00.r - ID rank 0
CryptoObject00.g - Coverage for ID rank 0
CryptoObject00.b = ID rank 1
CryptoObject00.a = Coverage for ID rank 1

Layer 2

CryptoObject01.r - ID rank 2
CryptoObject01.g - Coverage for ID rank 2
CryptoObject01.b = ID rank 3
CryptoObject01.a = Coverage for ID rank 3

Layer 3

CryptoObject02.r - ID rank 4
CryptoObject02.g - Coverage for ID rank 4
CryptoObject02.b = ID rank 5
CryptoObject02.a = Coverage for ID rank 5

Metadata in the file stores the original name of the elements and the hash method used along other information.


Matte

Given a name of an object/material/namespace we can compute the ID using the hash function documented in the metadata. All that is necessary to create the matte is to iterate through the ID coverage pairs and retrieve the coverage per pixel. If a single matte for multiple IDs is requested the coverage of the IDs is summed.


Answering your questions

What are levels?

Levels are the number of ID-coverage pairs per pixel. Therefore the levels determine how many unique objects can be differentiated per pixel. It should be a multiple of two, since levels divided by two is the number of layers in the Multi-layer OpenEXR.


Why are there three input sockets on the Cryptomatte node?

The three sockets are for the recommended three layers to store the six ID-coverage pairs per pixel. You can add additional layers in the sidebar in the Item tab in the Properties panel.


Why are the output sockets of the Render Layers named like that?

I believe this is an error, since they get properly named internally in BlenderSync::sync_render_passes() (blender_sync.cpp) however register_passes() (engine.py) does an increment by two. This is correct in regards to the number of render passes, since there are two ranks per layer. However the names should still use consecutive numbers and match the ones of the Cryptomatte node.

I've created a post on DevTalk to discuss this with the developers.

Update: The numbering has been fixed by commit rB3f3d1ad4800c6d5e9d1a45912841b118cbac9f39


Why should I connect more than the first render pass and why does it work when only connecting the first?

It depends how many unique objects you'd like to differentiate per pixel. If you connect the first render pass, then the cryptomatte node can only create a matte which contains the coverage of two objects per pixel at most, since you pass two ranks into it. This will likely be enough for most simple compositing tasks. For more challenging scenes with multiple objects that have motion blur and cross paths which each other, you'll want to use the additional passes as well.


References

Cryptomatte Specification Version 1.2.0

Friedman, Jonah, and Andrew C. Jones. "Fully automatic id mattes with support for motion blur and transparency." ACM SIGGRAPH 2015 Posters. ACM, 2015.

$\endgroup$
  • 1
    $\begingroup$ Thanks a lot for in-depth :) I just downloaded example bunny scene (more complex one) where functionality of levels is noticeable. Here is a gif if you find it illustrative dropbox.com/s/6ypxthsbjh0fwef/test_Crypto00%2C01%2C02.gif?dl=0 BTW_What the color next to "Crypto.." input socket works for? $\endgroup$ – vklidu Nov 4 '19 at 10:11
  • $\begingroup$ @vklidu The color next to the input socket is there simply because the socket type is RGBA and Blender always displays a color picker for these sockets, although it doesn't really make sense for Cryptomatte. BTW I had to fix the explanation of levels, which was wrong. Sorry about that it was very late when I finished writing the answer. $\endgroup$ – Robert Gützkow Nov 4 '19 at 10:28
  • $\begingroup$ I've written a patch for the socket naming. developer.blender.org/D6185 $\endgroup$ – Robert Gützkow Nov 4 '19 at 13:50
  • $\begingroup$ The patch has been accepted. developer.blender.org/… $\endgroup$ – Robert Gützkow Nov 4 '19 at 14:03
  • $\begingroup$ Thanks a lot for all your work :) Just still from end-user perspective I'm not sure if there is clear connection between "Levels" and node sockets with numbering like now - Level 2 / socket 00, Level 4 / socket 01, ... if I understood it right -levels counts objects / sockets just names layers, that make sence if you understand system behinde. But since layer represents some level, there would be nicer to let know user how they correspond to each other. Like for me seems more natural Level 1 (described as two unique object per pixel) is represented by socket named "Crypto 01". $\endgroup$ – vklidu Nov 4 '19 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.