I'm trying to use a mask to hide a part of a render layer, so that when the render layer is shown over a movie, some parts in the movie will appear to be in front of the render layer.
For example, assume I want to add a box to this image, so that it would look like it's behind the person:
I created the following simple object:
If I render the object, it looks great - the object is solid, everything else is transparent (I didn't use any 'sky'):
Now I create a mask to describe the standing person (or at least the part that would otherwise be covered by the box):
What I want now is to create a "hole" in the rendered box, so that when I use Alpha-over, the box look as if it's behind the person, but I can't find an easy way to do that. My first guess was to use 'Set alpha' to apply the mask (inverted) as an alpha for the rendered layer, but this doesn't look good:
The mask did create a "hole" where it should, but now all the other pixels are not transparent. How can I "retain" the transparent pixels while setting more pixels to be transparent based on the mask (use it as a matte)?
Note: I can do this with (I'm ashamed to say) some math work on the alpha channel, but it just doesn't seem right:
Thanks, and sorry for the [oh so] long question.