8
$\begingroup$

I'm trying to use a mask to hide a part of a render layer, so that when the render layer is shown over a movie, some parts in the movie will appear to be in front of the render layer.

For example, assume I want to add a box to this image, so that it would look like it's behind the person:

Standing person

I created the following simple object:

Simple box in Blender

If I render the object, it looks great - the object is solid, everything else is transparent (I didn't use any 'sky'):

Rendered box

Now I create a mask to describe the standing person (or at least the part that would otherwise be covered by the box):

Mask over person

What I want now is to create a "hole" in the rendered box, so that when I use Alpha-over, the box look as if it's behind the person, but I can't find an easy way to do that. My first guess was to use 'Set alpha' to apply the mask (inverted) as an alpha for the rendered layer, but this doesn't look good:

enter image description here

The mask did create a "hole" where it should, but now all the other pixels are not transparent. How can I "retain" the transparent pixels while setting more pixels to be transparent based on the mask (use it as a matte)?

Note: I can do this with (I'm ashamed to say) some math work on the alpha channel, but it just doesn't seem right:

Correct compositing, but complex

Thanks, and sorry for the [oh so] long question.

Improve this question
$\endgroup$
2
  • $\begingroup$ What doesn't look right about it? $\endgroup$
    – gandalf3 ♦
    Nov 26, 2013 at 20:46
  • $\begingroup$ It just looks like a lot of nodes for a very basic operation. Note that I used split RGBA just because I simplified an existing node graph, which included a blur node after the rendered layer node. $\endgroup$
    – Barak
    Nov 27, 2013 at 6:36

4 Answers 4

Reset to default
8
$\begingroup$

In your first attempt you are replacing the original alpha with the mask, you need to combine the original alpha (the render) with the new alpha (the mask):

enter image description here

Which would result in:

enter image description here

Based on this render and mask:

enter image description here enter image description here

That's essentially what you're doing in your solution that doesn't 'seem right'. This is just a slightly simpler way of doing that.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Is Subtract the right operation? What's the most correct math operation? Maybe it's multiply? I'm assuming alpha is in the range 0..1. $\endgroup$
    – Barak
    Nov 27, 2013 at 6:38
  • 1
    $\begingroup$ @user1145012 Subtract is the correct operation, we are subtracting white (mask) from white (alpha) to produce black (transparent) or 1 - 1 = 0. $\endgroup$
    – Ray Mairlot
    Nov 27, 2013 at 14:23
  • $\begingroup$ It's definitely right for full white and black pixels, but what if I had some fuzziness at the edges, how would 50% transparent and 50% transparent combine? $\endgroup$
    – Barak
    Nov 27, 2013 at 20:10
  • $\begingroup$ If you had a feathered mask (semi-transparent edges) then in this case the cube would also have fuzzy, semi-transparent, edges. $\endgroup$
    – Ray Mairlot
    Nov 28, 2013 at 18:04
  • $\begingroup$ @Barak If this answer helped, you can mark it as accepted by clicking the tick icon next to the answer. $\endgroup$
    – Ray Mairlot
    Apr 2, 2015 at 17:40
4
$\begingroup$

Here are a couple more ways of doing this:

enter image description here

And:

enter image description here

Improve this answer
$\endgroup$
3
$\begingroup$

How about something like this: node setup

Mix the footage and the CGI using the (inverted) mask as a factor.

This is the result (I didn't have any live footage on my computer... so here's a red cube added to SSBB): result

Improve this answer
$\endgroup$
2
$\begingroup$

Here's an example of a mask cutting out the sphere behind me, but what if I want it to be removed?

Mask filled hole

You can reverse the the mask opacity by simply clicking the little black/white icon in the mask tool properties.

Mask hole reversed

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .