OpenGL masking. (the speedier option)
Conventional masking will cut a hole in whatever's behind the mask, except when rendering in OpenGL.
OpenGL masks however can discriminate by whatever is the younger object, the mask or the item(s) to be masked.
Objects younger than the mask appear to be unaffected whereas those items created earlier than the mask disappear when behind it.
The masks themselves are simply 2D planes with their material slot-->Alpha (opacity) set to 0.008.
Both specular sliders should be at zero to avoid 'ghosts'.
The Diffuse setting should be at about half or less for the same reason.
The mechanics are simple enough - if set above the default alpha threshold value, Blender treats the mask as opaque, even though it's invisible when viewed in "Material View Mode".
Anything that gets behind it is therefore hidden from view.
To overcome the effect for selected objects, e.g. your textured background plane, you can parent the mask to the background and unparent it again immediately after. (child = mask) Make sure to select "Keep Transform" to avoid animation and origin point issues.
If parenting becomes an issue, you can instead, duplicate the background object, delete the original and use the copy. It will be younger than the mask and won't be hidden.
To reverse the effect, that is hide objects that are not being hidden, parent in reverse. (child = object)
Unparent again.
Parenting issues again? - duplicate the mask, delete the original, use the copy.
To view the result, get into Object mode, use "Material View mode" and always press A to deselect everything.
Render in OpenGL
NOTE -
Should you wish for better background image quality, don't texture a background plane, instead shape the mask using the "Background image" option in the 3D window's properties. (press N)
Wind it's opacity up to full.
Set the render-output "shading" to "Transparent".
Set the output to png files with the RGBA button pressed.
Render the video out in OpenGL and without the background image.
All areas that are not 3D objects will be rendered transparent. An object moving behind the mask will also go transparent.
Mix this image sequence with the background video in the VSE as described immediately below...
-============================-
Blender render masks -
Blender provides masking but it isn't discriminatory and will want to cut a hole through non-masking layers.
To set one up, shape the mask plane, place it in layer 2.
Place the objects to be masked in layer 1.
Go to the layer properties at right. (next to the right of the camera icon)
Open the Layer tab.
In there will be 3 sets of layer charts (20 in each) that can be checked (dark) or unchecked (light)
The set at the left called "Scene" are all the layers that are to be included in the render. If we use layer 1 for non masking objects and layer 2 for the mask, then check both layers. (1 & 2)
At the right of that is the "Layer" set. In there, only check layers that are not masks. (layer 1)
Underneath that is the masking set. These are layers that are going to mask out the others. (layer 2)
Any object in layer 1 that gets behind an object in layer 2 will be hidden. The masking object itself will be rendered invisible.
-================-
Because we need to discriminate, we'll need to firstly render out this video with an alpha layer so the transparency is included.
Set the Output properties -->"Shading" tab to "Transparent". This will make everything except visible objects transparent.
Set the rendered output to png files with the RGBA button highlited. A set of files, one for each frame will be created in the output folder.
VSE -
The resulting png files can be imported in the VSE as an image sequence and treated like any other video file. SH-A-->"image"
Set this strip to "Alpha Over" in the properties at right. (press N)
With our video at the upper strip and background image or video in the lower strip, the transparent areas (and masking holes) will allow the background to show though.
As an object moves behind a properly shaped mask, it will progressively disappear and look as though it has just moved behind an item in the background video.
-================-
Chroma key - (green screen)
This blanking is achieved using a keying node in the compositor. The masking and non masking objects are rendered with the green screen behind them. The green video isn't alpha layered.
The masking object has to be coloured the same green as the background green. As the object to be hidden moves behind the green mask, it is of course hidden and won't be rendered while it's there.
The mask will also be invisible by virtue of it's colour.
The rendered video is later presented to the compositor's keying node which converts the green to transparency. This is how the mask becomes transparent and anything that's not green, doesn't.
This output is then connected to an "Alpha Over" node which does exactly as does the mixing in the VSE example.
The setup to make this happen -
The output from the keying node will be transparent wherever it sees the right colour as set in the little colour window at it's lower end. This is put through an "Alpha Over" node to do as we did in the VSE.
Show the background image under the 3D objects.
To mask out an object so it moves 'behind' an item in the background image, shape your mask to match the item's shape and place it between the camera and the object to be hidden.
-==================-
Compositor ID masking (?) and Cycles - No experience with those.
