7
$\begingroup$

I searched around Blender SE, and I have seen no questions down this slant at all, so I'm going to ask it.

Blender uses all the information necessary to successfully solve a track to re-position a camera in 3D space in relation to the tracked footage.

My question is, since all of these "Control Points" get solved to some high degree of accuracy, and become a known distance and angle to the camera, is there any quick way to actually build a mesh from the video track once it is solved (allowing the rest of the texture to become auto-trackable based on commonality between frames - maybe something similar to how microsoft kinect gets used for 3D scanning, but as a post process after the solve of course)?

To maybe add more references of my thought process into something (at hopefully the most simplistic level) most could follow (as you jump into my rabbit hole of theoretical thinking), it would be something like the following image portrays.

enter image description here

If I imagine this fast wiggle as an accelerated fast-forward/rewind view of the footage & solve, my own brain is solving the 3D of this image (even if I close one eye - unlike stereo view).

This leads me to believe that Inverse to a stereo camera (such as a kinect/artec snanner/ etc... - which uses at least two camera's to form a triangle to millions of points in space, and solve their positions) you would have multiple basically solved locations in the space triangulating back to at least a single camera (eg. my one eye open that is pretty stationary OR both eyes open which helps average the solve in my brain I guess, Ultimately the Blender Camera(s)). This effect works so well, because it acts as a POV (persistence of vision - not referring to point of view) to the brain. It just so happens that Blender has this information in the next & previous frame locations in the form of pixel data of the footage, so what's the technical difference in terms of visual memory/persistence?

Also this image is very clean in terms of focal subject and little occlusion, except for at the very perimeter of the sphere. All the rest of the Image Texture is clean and trackable (hopefully with some sort of automatic help in the future).

I've tried this visually/manually from two images, and got some pretty amazing results, but it requires manipulating the mesh in such a way to where the topology in true 3D space & the UV's in 2D space would form the solve to look good.

Manual_Visual 3D Solve

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ There are no such tools in blender. Related: blender.stackexchange.com/questions/41813/… $\endgroup$
    – user1853
    Mar 18, 2016 at 3:11
  • $\begingroup$ also related: blender.stackexchange.com/questions/14164/… $\endgroup$
    – user1853
    Mar 18, 2016 at 3:17
  • $\begingroup$ @cegaton thanks for the links. The first one is closest to what I am looking for, but in a more automatic way. Once you have tracked points you should be able to find as many commonalities between frames as the pixel info provides and automate the uv map from camera view as well. $\endgroup$
    – Rick Riggs
    Mar 18, 2016 at 4:32
  • $\begingroup$ Without being able to properly articulate what I'm after, I see this very similar to how we can currently do a z-pass render but the exact inverse. Does that make sense? A whole bunch of pixel data, a camera distance, and a 3d mesh. Like the components here $\endgroup$
    – Rick Riggs
    Mar 18, 2016 at 4:32
  • 1
    $\begingroup$ If anyone reads this while writing a script that does this (or something similar), we are talking about multiple frames, so when baking the image-sequence to actual textures for the object(s) built from the point-clouds from tracking, maybe weight the pictures according to each face's normal alignment to the camera vector at that frame. $\endgroup$
    – WorldSEnder
    Apr 7, 2016 at 0:19

2 Answers 2

Reset to default
5
+50
$\begingroup$

There is currently no way to reconstruct a more detailed mesh between the camera-solved Control Points from within blender. Thats because the Camera Solver does not look at the image data at all, It only deals with the marker positions. The Control Point-Tracker is the one who finds commonality between frames. So you could just add more Control Points, but you'll of course never get a z-value for every pixel of the frame.

So I see this two ways of achieving, what I think you want:

Manual Scene Reconstruction - this works well with rectangular scenes like rooms, hallways, roads, houses

Blender Movie Clip Editor offers a few basic tools for scene orientation after camera solution: automatic origin, floor or wall plane setup from selected tracks. With this you can line up Blenders coordinate-grid with rectangular objects in the movie clip. In a camera-viewport with the movie-clip enabled as background its easy now to manually reconstruct rectangular objects via extrude on the axes x,y or z. Also, the BLAM Blender addon may be helpful with this process. It also features camera solution from one single image instead of a movie clip.
In the end you can texture the mesh using the UV-projection modifier to project the movie onto the mesh.

Foto Series Reconstruction Workflow - also works with non-rectangular scenes and objects, like reliefs, uneven terrain

Extract ca. 25 frames with significant parallax from your clip (e.g. take every 10th frame), then...

Option A:
Use the free 123D Catch-App from Autodesk to generate the model and export it to Blender.

Option B:
Follow this video tutorial, which describes the workflow of reconstructing and texturing a mesh from a series of fotos using the free programs VisualSFM and Meshlab, and exporting the model to Blender. VisualSFM is used to reconstruct a 3d-point-cloud from a series of fotos, with Meshlab you approximate a mesh from the points and optionally create a UV-Map with projected parts of the fotos as texture. From there the model can be exported to Blender.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the input. I upvoted it because I appreciate both the attempt and the information that you provided. I know basically what you are talking about as i have been keeping an eye on these features. My main intent is to do a pass afterword to reverse triangle (traverse) between points (same point different frame for every recognizeable pixel) and the solved camera location, fstop distance, distortion, etc. Between this and the camera solve info it seems that all the info is there to do the SFM (structure from motion) operations. $\endgroup$
    – Rick Riggs
    Mar 18, 2016 at 4:19
  • $\begingroup$ As far as Blender not looking at the pixel data, I believe that this is false to a degree, because once you set a point to track, you give that point an area window (limit area) to find your pixel in the next/prev. frame(s). So there is something analyzing the pixel data. The primary thought in my mind, is once the trackable points are solved, then the rest of the points (at least within the degree of error of the solve) stay in a fairly relative relationship to each other, given that there is foreground, midground, and background objects that move at different rates & occlude others. $\endgroup$
    – Rick Riggs
    Mar 18, 2016 at 6:10
1
$\begingroup$

I believe you are looking for something like this: https://www.youtube.com/watch?v=GEAbXYDzUjU . It uses a few third party softwares (totally free) to construct meshes from pictures you take of a real-world object. Better yet, it does all the tracking and data sorting for you, so all you have to do is take the pictures and input them into the programs. Hope that helps!

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .