14
$\begingroup$

I'm having this issue for years now, and I'm not sure if I am doing something wrong on my end, or if what I would like to achieve - or expect to see - is even possible. My question is about compositing artifacts when using Blenders mask layers. Let me explain:

In the Blend file provided, there are two objects: Dragon A and B. Each of them is on a separate layer, rendered on a separate render layer, and masked with the other one. So A is masked out by B, and B is masked out by A. Like this:

scene setup

scene setup

RenderLayers A and B

RenderLayer ARenderLayer B

When I render, I write out two MultiLayer EXRs with 16bit depth. That Stuff, I feed into the following comp:

comp

and it gives me this:

white lines

You can clearly see the white lines around the areas where the dragons mask each other.

If I comp in a weird fashion like this:

weird comp

I get this:

result

It's not perfect, but still a lot better than the first one. What I am doing is kind of cutting off the bright alpha values, making a 0.75 alpha value equal 1.

My question is: Why do I see the white background shine through after all? Is it something which is fundamentally unchangeable and we have to live with? Am I doing something wrong on my side? Do you have an alternative solution?

Edit

The solution Ray Mairlot came up with solves the issue by only masking one of the two, the other one would exist un-masked underneath, and thus provides a perfect overlay. However, I'd like to still mask one against the other for specific reasons: I'm sending images to a web application, which layers them dynamically (a configurator in essence). There can be 30 or more image layers, each of them needs to be downloaded by the website visitor. If I could mask one part against the other, the individual file size would be smaller. Not really a necessary reason, but that's why the idea came up. The other reason is, if I mask the layers, the individual layers render faster. This can save me some 30% of the render time per configurator (which renders on average for 12 hours).

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Did you try checking the "convert premultiply" on the Alpha Over node?? $\endgroup$
    – Samir Rahamtalla
    Commented Nov 16, 2016 at 16:17
  • $\begingroup$ @SamirRahamtalla I did, in this case it makes no difference. I also tried reordering the composition, so first the dragons would be comped, and then I'd comp against white background. No difference. $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 16:18
  • 1
    $\begingroup$ +1. This is a good question. A very good question. It is Specific and Clear. It shows what you've tried, what you think, and has useful images. I wish this site had more questions like this. $\endgroup$
    – X-27 is done with the network
    Commented Nov 16, 2016 at 16:55
  • $\begingroup$ "Why do I see the white background shine through after all?" The back image can't block the background because it's masked by the front image. The front image is not fully opaque everywhere, so doesn't completely mask the background on its edge. $\endgroup$
    – David Schwartz
    Commented Nov 16, 2016 at 20:22

4 Answers 4

Reset to default
17
$\begingroup$

Those two images you're trying to composite together have "correlated" alphas. The Alpha over operation isn't the appropriate operation for this situation.

The solution is simple: you just have to add those images together, then add their alphas together and assign the resulting alpha to the result of the first addition.

Here's how:

Compositing two images with correlated alphas

Why alpha-over doesn't work

Alpha over is, like its name states, an "over" operation. Something is over something else, covering it. The alpha channel determines the occlusion of the background by the foreground.

When you inspect the alpha-over formula for two associated (premultiplied) images, you'll see it's just an addition, but before that addition the alpha channel of the foreground is inverted and multiplied on the background, casting a shadow on the pixels that the foreground occludes.

Associated Alpha Over Operation:

Foreground.RGB + ( ( 1.0 - Foreground.Alpha ) * Background.RGB )

Porter Duff Alpha Merge Operation Uncorrelated Geometry:

Alpha.A + Alpha.B - ( Alpha.A * Alpha.B )

Correlated Geometry:

Alpha.A + Alpha.B

When alphas are correlated, like in your example, there is no foreground-background situation. It's rather filling a hole in one image with the other image.

The hole and the image that will cover it fit perfectly, so there is no reason for casting a shadow on the background. There's already a hole in it. If you use alpha over, the "shadow" of the foreground will be multiplied again on the antialias pixels that are already shadowed by the mask, causing a doubled value of transparency, and that's why you get those semi-transparent pixels where they should be completely opaque.

So, skipping the "over" part by just adding the two plates together works perfectly. In Blender you have to add alphas too and assign the result to the mix.

Improve this answer
$\endgroup$
5
  • $\begingroup$ This works like a charm! Would you mind adding a reference to the Alpha Over formula you mention, or insert the formula directly into your answer? This way it might become clearer as of why this solution works, and the other one doesn't. Two years I've spent wondering about this issue, can't believe the solution is that simple... $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 20:20
  • $\begingroup$ I would add that thinking about it in terms of probability (remember that alpha in associated alpha imagery is geometry) that we can see A+B - (A*B) turns the latter half into 0 when the geometry "perfectly snaps together". Hence there is no cheating nor hacking here, but rather the same concept considered against the geometry of the original situation, slightly hidden in the fact that the rasterized pixel is blending the geometric result. $\endgroup$
    – troy_s
    Commented Nov 16, 2016 at 20:44
  • $\begingroup$ en.wikipedia.org/wiki/Alpha_compositing Think of alpha channel as a model of emission and occlusion: an object in front of another adds its own light, but also occludes the light coming from the object behind. When you perform an alpha-over operation, it has two parts: it inverts the alpha and multiplies over the background, creating a holdout (that's the occlusion part) Then the foreground is added (that's the emission part). The latter is what we have to use in this case, just the added emission. $\endgroup$
    – Gez
    Commented Nov 16, 2016 at 20:48
  • $\begingroup$ The alpha over formula for each pixel is: Co = Ca + Cb × (1−αa) for colour and αo = αa + αb × (1−αa) for alpha channel. $\endgroup$
    – Gez
    Commented Nov 17, 2016 at 21:04
  • $\begingroup$ Another question related to this. blender.stackexchange.com/questions/75953/… Can this solution be extended to many render layers? $\endgroup$
    – bot4u
    Commented Sep 27, 2017 at 7:15
3
$\begingroup$

From what I can see, the error is that your original render layers from your 'Render' scene are set to mask each other out. I can't personally see a reason why you would mask them (particularly not both of them) if you are looking to combine them.

Disabling the masking on the renderlayers:

enter image description here

And using this node setup:

enter image description here

Gets this:

enter image description here

Improve this answer
$\endgroup$
5
  • $\begingroup$ That's a very good advise, +1 for this. In the real world example, I'm sending images to a web application, which layers them dynamically (a configurator in essence). There can be 30 and more image layers, each of them needs to be downloaded by the user. If I could mask one part against the other, the individual file size would be smaller. Not a really necessary reason, but that's why the idea came up. The other reason is, if I mask the layers, the individual layers render faster. This can save me some 30% of the render time per configurator (which renders on average for 12 hours). $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 17:05
  • $\begingroup$ But the main reason is: I DON'T GET WHY IT SHOULDN'T WORK :)) $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 17:06
  • $\begingroup$ I don't really understand why it would work, but I don't have the knowledge to explain why. I imagine it is to do with each of the objects having partially transparent pixels at their edges leaving a transparent line inbetween them. $\endgroup$
    – Ray Mairlot
    Commented Nov 16, 2016 at 17:26
  • $\begingroup$ Also, I would suggest you add in to the original question that you need each layer to be masked by the other. $\endgroup$
    – Ray Mairlot
    Commented Nov 16, 2016 at 17:35
  • $\begingroup$ I've edited my question as you suggested. Meanwhile, I did some tests in Gimp. I'll try to replicate that tomorrow in Photoshop too, just to make sure, but if I get the same results on that simple test (not the dragons, something different), then you're most likely right: It's impossible. $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 18:52
1
$\begingroup$

You don't need two Aplha Over nodes. This way it works: enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer, but the problem is: I need to see my comp against white, as the final output will be on a white backplate (studio shot). The white background was not added by accident. $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 15:25
  • $\begingroup$ I think the problem is in the render, because if I open them in Photoshop and layer them on top of each other there is the same issue. $\endgroup$
    – float
    Commented Nov 16, 2016 at 15:35
  • $\begingroup$ I see the same issue, also when we comp in AfterEffects or Nuke. But why is it behaving like that? It might be that this effect we see is totally valid and makes sense (from a math point of view), but I haven't been able to figure out why. $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 17:08
-1
$\begingroup$

The problem is that you're overlaying the silver dragon with a white background, before you overlay it on the orange dragon. Just use one alpha-over. One image with an alpha channel goes into one slot and gets overlayed onto another image which goes into the other slot, instead of the white color.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the answer, but the problem is: I need to see my comp against white, as the final output will be on a white backplate (studio shot). The white background was not added by accident. $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 15:26
  • $\begingroup$ not sure why this has been downvoted, thanks for the answer still $\endgroup$
    – aliasguru
    Commented Nov 16, 2016 at 20:56
  • $\begingroup$ (I'm not sure either... oh well) In that case, you should add the white back plate last. Use one alpha-over to combine the two images, then use another alpha over to put the image over the white. $\endgroup$
    – Matt
    Commented Nov 17, 2016 at 14:12
  • $\begingroup$ I've tried that already, makes no difference. The answer which Gez gave however is absolutely correct. It has to do with the way alpha operations are mathematically defined, and Alpha Over in my case was simply a no-go. $\endgroup$
    – aliasguru
    Commented Nov 17, 2016 at 14:55
  • $\begingroup$ Interesting... Glad you got it sorted, anyhow. $\endgroup$
    – Matt
    Commented Nov 18, 2016 at 21:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .