White lines appearing when rendering shadows to separate layer and recombining in compositor

Question

I have trawled forums for ages looking for a possible answer to this.

I am trying to render shadows to a separate layer then recombining them in the compositor with the rest of the image. This is because I would like to have complete control over the shadow layer and ultimately apply a mask to it (to avoid any shadows extending past the render border creating a hard edge).

I have created a simplified setup to demonstrate the issue.

In this simplified setup I have 2 view layers, layer 1 containing a cube and layer two containing a shadow catching plane. You can see that the lines appear at the border between the shadows and the object. I have looked into issues of premultiplied alpha but none of this seem to sort it. They appear despite the condition of either Convert Premul checkboxes on the Alpha Over nodes.

I suspect that it may be to to with the Transparency filter in the Film section of the properties, however altering this leads to poor results.

Does anyone have a solution or some tips on what to investigate. This will ultimately be for an animation and so I'm keen to achieve everything in Blender rather than saving 2 sets of frames for the shadows and for the objects

For reference here is an image of just the first view layer with both the cube and shadow cater being rendered showing no lines

Any help or advise would be massively appreciated

Answer 1

I did a test scene like the one you did. I thought it was due to the premult setting (usually that's the guilty party for compositing edges), but that's not the case here. Part of it is due to anti-aliasing and the pixel filter - not because of bad settings, but just the nature of it.

With the cube by itself, the edges of the alpha are going white to black, with some grey in the middle. For the shadow catcher and holdout cube, the edges between the shadow's alpha are going white to grey to black, in the opposite direction. This edge is on the same pixels for both layers, of course. When you do an alpha over, that edge results in some grey alpha values from both layers competing for the same pixels. Therefore, some of your white background peeks through and shows up as white

The best solution is, simply, don't composite this. If you can avoid compositing layers of the same CG scene, render the shadow and object together. My guess is you want to do this for visual effects; I do that too and run into this situation often. I usually render the shadow together with any CG object I'm putting in a video to get a clean edge.

If you must composite them together, you can do some masking and simply expand the shadow layer's alpha edge with a dilate/erode node. That can get messy and give strange results, but it's possible.