Can dynamic lighting on meshes be translated onto a video clip?

Question

Is it possible to procedurally affect the appearance of a video clip depending on the luminance of an area of a mesh?

Question detail:

The process I'm trying to do is explained below.

• import and track a video clip (for ease of explanation, it is the corner of a room, as in the attached image)

• set up a 3D scene with the camera solve from the video clip via 'setup tracking scene'
• model the room corner in 3D
• the video is going to be used day for night, so parameters (brightness, contrast etc) will be adjusted to make it appear to be a night scene)
• dynamic lighting elements will be introduced in the 3D scene, which will cast light on the meshes used to model the room corner

What I want to do is use the luminance of the meshes to adjust the brightness (and other parameters - if there are coloured lights I would like to tint it for instance) of the video clip so it appears it is being illuminated by the dynamic lighting elements. Is this possible?

Answer 1

If you need interactive lighting you are going to need to recreate the geometry and project the lights into it, and combine that with the video background.

Note that to successfully reconstruct the geometry with motion tracking using the video as a source, you will need to move the camera in a way that shows parallax, a stationary camera will not work. If your camera is stationary and just rotates, reconstruct the geometry using fspy and match the camera move in blender using tripod solve.

Do a search to understand how a shadow catcher works.

The light from the video will have to be recreated in blender to affect whatever objects you have in the virtual environment and have the shadows and reflections of said objects affect the virtual geometry that will work for shadows and color. So make careful notes of how the lights are set up in the real world.

Also, keep in mind that you will need to mix several passes together to pull off something that looks realistic. The interaction of light, specially indirect light (that caused by the reflection of light on the surfaces of the objects in the scene) has to be carefully examined. For example, anything that moves will change the shape and color of the shadows and reflections. Pay special attention to the shiny surfaces on the lamp, as those will have to reflect real and virtual objects at different degrees, the metal and glass will behave differently.

The ratios of the lights in the real world and those in the virtual environment have to match as closely as possible. If you have access to a real camera (not a dslr or a phone) shoot using log encoding so that you can record values as scene referred and not as display referred. Having linear information will make the integration easier and more convincing.

In short you are in for a lot of fun if you decide to do this. Know that there are no real shortcuts, a shot like this requires careful planning and attention to detail.

Suggested links:

How to get reflections from 3D to show up on real images

How can I get better results when doing camera motion tracking?

Realistic rendering of mesh together with image?

Render shadows over a transparent background in cycles

Add color to shadow catcher

Cycles: Shadeless "Shadow-catcher" Material for Rendering onto Footage

Answer 2

Set on the right path by the answer above, for anyone else looking this is a good answer to what I was asking: Is there a way to utilize emissions in VFX?