Inserting green screen footage into 3D space

Question

So, the Piano Guys made a pretty cool video a while ago where they had green screen footage chroma keyed and then inserted into 3D space as some sort of plane like object. The process can be seen in this video :https://youtu.be/mJiEXKF6WTI

I understand how to do lightsaber composition (by superimposing a shadeless cylinder on top of footage and then blurring and coloring it, however if there is a more efficient way, I would like to know.), and I understand the basic process of chroma keying, but once these steps are completed, I am somewhat confused about how to place these video in a 3D setup like he does in the video (having this way as opposed to simply overlaying the footage on top of the renders allows for paralax and such). I would like to do this with preferably as few steps as possible.

I have an idea on how I might do this, but I would like input from the community so I don't waste as much time. I'm thinking maybe to key out the green screen, export png sequences with transparency for every piece of footage, and then importing images as planes into a 3D scene, however this would not be very effective if I had lots of clips and I had to do the same thing for each one. Preferably, I would like to see the footage in the scene in real time (although the scene doesn't have to look rendered) while animating the camera and such.

Thanks a bunch!

Answer 1

Probably the best way, is to key out the green, and then import the image sequence into the 3D view.

Start by taking the image sequence into Blender, or your compositor of choice, and key out the green to your taste. Try to keep in mind the environment it's going to end up in and make sure that the key will reflect that. Then render out the frames as a PNG sequence, or any other image file format that Blender accepts that uses transparency.

In Blender, use the import image as planes addon (included by default) to import the image sequence into the 3D view. Make sure that alpha is enabled, and shadeless is checked in the settings before importing.

(You want to have shadeless checked, because if it interacts with light, it will be treated as a flat object, so any shadows that are there will be completely flat.)

Then place the plane where you want it to be, and use the color tools to make it fit into the environment properly. Or use some render layers magic to render it out separate from everything else to allow you to composite it back in properly either useing Blender or an external compositor.