I'm having this issue for years now, and I'm not sure if I am doing something wrong on my end, or if what I would like to achieve - or expect to see - is even possible. My question is about compositing artifacts when using Blenders mask layers. Let me explain:
In the Blend file provided, there are two objects: Dragon A and B. Each of them is on a separate layer, rendered on a separate render layer, and masked with the other one. So A is masked out by B, and B is masked out by A. Like this:
RenderLayers A and B
When I render, I write out two MultiLayer EXRs with 16bit depth. That Stuff, I feed into the following comp:
and it gives me this:
You can clearly see the white lines around the areas where the dragons mask each other.
If I comp in a weird fashion like this:
I get this:
It's not perfect, but still a lot better than the first one. What I am doing is kind of cutting off the bright alpha values, making a 0.75 alpha value equal 1.
My question is: Why do I see the white background shine through after all? Is it something which is fundamentally unchangeable and we have to live with? Am I doing something wrong on my side? Do you have an alternative solution?
The solution Ray Mairlot came up with solves the issue by only masking one of the two, the other one would exist un-masked underneath, and thus provides a perfect overlay. However, I'd like to still mask one against the other for specific reasons: I'm sending images to a web application, which layers them dynamically (a configurator in essence). There can be 30 or more image layers, each of them needs to be downloaded by the website visitor. If I could mask one part against the other, the individual file size would be smaller. Not really a necessary reason, but that's why the idea came up. The other reason is, if I mask the layers, the individual layers render faster. This can save me some 30% of the render time per configurator (which renders on average for 12 hours).