Render Layers causing pixelation

Question

I used this answer (Prevent shadow on object/material A, allow for object/material B) and it helped me a lot. However, this method is causing these jagged black pixels when I am combining them. What gives?

Answer 1

The index passes in Cycles are only rendered with only 1 sample, unfortunately, so they are not antialiased.
Your setup looks to me as if it should work with a simple Alpha Over too. Replace your Mix node with AlphaOver and just don't use the index pass.
However, for that to really work correctly in every case you have to make sure that the object you want to separate is masked by the occluding geometry from the rest of your scene.

So put the object that you want to separate on a layer with only that object on it. Then in Renderlayers exclude the layer with the element that should not appear on that layer by using the Exclude Layers, like you already did. Finally set the Mask Layers to all layers with objects on it, that would occlude the separated object. In your case that would be the table, in case the makeup thing would penetrate the table geometry a bit. Though in your scene you might even be lucky enough that it might work without even bothering with Mask Layers.

If you really have to use an index pass you can try to smooth it with the Dilate/Erode node, with mode set to "Feather".

Answer 2

First try enabling the use alpha option on the mix node (the little icon at the top right of the node). If that doesn't work there are two other relatively easy things to try:

1. Slightly blur the output from the ID Mask node so that there are no hard edges.

2. Use the separate RGBA node after the second render layer. Add a combine RGBA node rights after it and connect all of the inputs. Then use the dilate/erode node on the alpha in between the two nodes just added and set the distance to a small negative number to remove the pixels along the edge.