I'm trying to use a mask to hide a part of a render layer, so that when the render layer is shown over a movie, some parts in the movie will appear to be in front of the render layer.

For example, assume I want to add a box to this image, so that it would look like it's behind the person:

Standing person

I created the following simple object:

Simple box in Blender

If I render the object, it looks great - the object is solid, everything else is transparent (I didn't use any 'sky'):

Rendered box

Now I create a mask to describe the standing person (or at least the part that would otherwise be covered by the box):

Mask over person

What I want now is to create a "hole" in the rendered box, so that when I use Alpha-over, the box look as if it's behind the person, but I can't find an easy way to do that. My first guess was to use 'Set alpha' to apply the mask (inverted) as an alpha for the rendered layer, but this doesn't look good:

enter image description here

The mask did create a "hole" where it should, but now all the other pixels are not transparent. How can I "retain" the transparent pixels while setting more pixels to be transparent based on the mask (use it as a matte)?

Note: I can do this with (I'm ashamed to say) some math work on the alpha channel, but it just doesn't seem right:

Correct compositing, but complex

Thanks, and sorry for the [oh so] long question.

  • $\begingroup$ What doesn't look right about it? $\endgroup$ – gandalf3 Nov 26 '13 at 20:46
  • $\begingroup$ It just looks like a lot of nodes for a very basic operation. Note that I used split RGBA just because I simplified an existing node graph, which included a blur node after the rendered layer node. $\endgroup$ – Barak Nov 27 '13 at 6:36

In your first attempt you are replacing the original alpha with the mask, you need to combine the original alpha (the render) with the new alpha (the mask):

enter image description here

Which would result in:

enter image description here

Based on this render and mask:

enter image description here enter image description here

That's essentially what you're doing in your solution that doesn't 'seem right'. This is just a slightly simpler way of doing that.

  • $\begingroup$ Is Subtract the right operation? What's the most correct math operation? Maybe it's multiply? I'm assuming alpha is in the range 0..1. $\endgroup$ – Barak Nov 27 '13 at 6:38
  • 1
    $\begingroup$ @user1145012 Subtract is the correct operation, we are subtracting white (mask) from white (alpha) to produce black (transparent) or 1 - 1 = 0. $\endgroup$ – Ray Mairlot Nov 27 '13 at 14:23
  • $\begingroup$ It's definitely right for full white and black pixels, but what if I had some fuzziness at the edges, how would 50% transparent and 50% transparent combine? $\endgroup$ – Barak Nov 27 '13 at 20:10
  • $\begingroup$ If you had a feathered mask (semi-transparent edges) then in this case the cube would also have fuzzy, semi-transparent, edges. $\endgroup$ – Ray Mairlot Nov 28 '13 at 18:04
  • $\begingroup$ @Barak If this answer helped, you can mark it as accepted by clicking the tick icon next to the answer. $\endgroup$ – Ray Mairlot Apr 2 '15 at 17:40

Here are a couple more ways of doing this:

enter image description here


enter image description here


How about something like this: node setup

Mix the footage and the CGI using the (inverted) mask as a factor.

This is the result (I didn't have any live footage on my computer... so here's a red cube added to SSBB): result


Here's an example of a mask cutting out the sphere behind me, but what if I want it to be removed?

Mask filled hole

You can reverse the the mask opacity by simply clicking the little black/white icon in the mask tool properties.

Mask hole reversed


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.