A good way to render objects on a photo background that works also with focus-blurred margins

Question

I'm trying to render with Cycles on a photo background using Blender Guru's compositor nodes:

[ Here, the foreground objects are obId=1 and are on layer 1 (rendered in RenderLayer); reconstructed background objects are obId=2 and are on both layer 1 and 2. Layer 2 is rendered as "ground". ]

However, in the current frame the main subject is supposed to be out of focus. As a result, both the ObIndex=1 pass (foreground) and the ObIndex=2 pass (reconstructed background objects, i.e. shadow catchers) are blurry. See the first two panels below:

As you can see, the blurriness doesn't consist in a smooth ramp of fractional intensities; it is instead a random cloud of blacks and whites, which will give an unusable result in terms of foreground/background separation (see the third panel).

I guess this is because the ObIndex pass only contains one integer per pixel, that can be either one of the "Pass Index" values assigned to the objects in the scene, but not anything in between.

So, my question: is there a better way to blend a rendered foreground onto a photo background that works well with blurred objects?

---

Here's a blend file with the scene described above.

Answer 1

As 3dpointedit pointed out, there is a limitation of the current ID mask system:

Is Z-Render Pass accuracy dependent on sample count? Are others like object id? (Cycles)

As such, these layers are unusable for any post-processing workflow. You need to obtain these layers differently, or build the pipeline in another way.

The correct way of compositing onto a background is detailed here:

Cycles: Shadeless "Shadow-catcher" Material for Rendering onto Footage

In short, you need to put your objects in layer 1, your shadow catchers in layer 2, and the lights in both layers. The render layers will be "object only" (include: 1), "shadows" (include: 2), and "clean background" (include: 2, exclude: 1). Render with transparent background.

This will give you these layers:

• "shadows"

  

• "clean background"

  

Dividing the former with the latter you get the shadow:

You multiply this shadow with your backdrop, and you alpha-over the object into your backdrop. You have to render the scene 3 times though.

You can also use the new Shadow catcher feature of Blender 2.79, but it will only produce black shadows.

Answer 2

I've found a workaround but real solutions are still very much appreciated!

This workaround is far from being optimal, as it depends on several parameters, it works only at Render Size = 100% (I'm not sure why the Gaussian Blur at smaller sizes creates light artifacts), it blurs the background as well as the subject, and it blurs by 1-2 pixels also borders that are supposed to be sharp.

---

Blurring the noise out

This is obtained by blurring in post-production the area where the ObIndex mask is patterned, that can be detected by blurring it and finding non-integer regions.

1. The yellow box is the same as in my question, taken from a Blender Guru tutorial.

2. The red box is a Gaussian blur of the final composite.

3. The Gaussian blurred image and the image itself are mixed using a Mix node with a Fac created in the green box. The latter is obtained by manipulating the ObIndex=1 ID mask. Zoom:

   

   This is what the result of the green node looks like: