Projection painting from images sequences/movies

Question

I have a shot where the camera is moving around an object. The scene was tracked and a digital object was modeled over the real one. I want to use the image information from the movie clip to paint over the recreated object (on a texture) by stepping through the frames and using the current frame as my projection. (Hopefully my explanation makes sense)

So far I noticed that in the source clone image options from the clone brush in texture paint mode, I can use the viewer node. Maybe that could do it?? So far I wasn't successful.

How can I make this work?

Answer 1

As nobody seemed to be able to answer my question, here is how I worked around my problem:

So my goal was to paint a texture over an object, using image information from a movie clip (displayed in the background image as a camera clip).

My first approach was trying to use the clone brush in texture paint mode. The problem here is that it is only possible to use an image as the source. I was then thinking about writing a python script, that would use the viewer node to export an image (current frame in camera clip), load it into the source from the clone brush so that I could essentially paint on my object from the camera clip. But ... yea that turned out to be too complicated and too long for me to work on (would be a nice feature though). So I looked for a more reasonable and not so time consuming solution.

The thing I came up with seems pretty straight forward. Simply split up your movie into chunks and use one image per chunk. Then I could make vertex groups, unwrap them using camera projection and project the image that I want for that specific group onto the texture. I then combined the textures and smoothed the edges out, so that you wouldn't see them in the final result.

And that's it. I think the clone brush would have been a nicer solution, but this also worked fine.

Hope this helps someone out there :)

Answer 2

I had the same problem: I had 3D tracked footage as background in camera, and wanted to project paint over reconstructed geometry. Ideally one could set the same clip as in camera as brush texture, but it doesn't seem to work that way. I solved this by exporting footage as jpg sequence, then importing single jpg as a source for brush texture and then painting over reconstructed geometry. I set it do display as stencil and fit it in camera, but I don't know how to do it precise. I did it few times when camera angle changed. It's an ok solution if you are using shadow catcher and want the surroundings to reflect in placed object in your motion tracked footage.