Compositing video footage into a 3d scene?

Question

I am trying to composite some footage of a person that has been chroma keyed into a scene of a dark room with flickering lights. My problem is that I don't even know how I would get close to making it seem somewhat realistic.

How would the lights affect the person if they aren't actually a 3D model, just a flat video? For reference, this video is basically what I am trying to replicate, minus the robotic camera movements.

Video Link

Answer 1

It will depend on a number of things, not the least of which is your footage device. If your footage device can capture a log encoding, there’s a trivial way to get from the camera code value encoding to the reflectance level and colours that the camera saw.

In order, you will loosely need to consider the following:

1. Illumination of the scene. Any CGI will need to have the illumination / emissions of the scene pushed back into the CGI context. HDRIs or proper camera footage captured from the location can help with this, with further work.
2. Tracking. If the camera moves at all, it will require tracking.
3. Distortion. The camera will have a degree of lens distortion that will need to be removed, then reapplied.
4. Proxy geometry. For certain emission effects and shadows, a loose idea of the scene will be required in terms of geometry.
5. Proper footage. Log encodings that capture scene reflectance is more or less mandatory here.
6. Decent camera rendering transform. The CGI and the footage will need to be composited under a reasonable camera rendering transform to make the effect as seamless as possible within the limitations of your toolset.

Paul Chambers has done a fantastic series of blog posts that leads down this rabbit hole, and is well worth researching. From using loose proxy geometry to exploring how scene referred lighting ratios work, his posts are tremendously insightful. There is a wealth of information hidden away over there.