8
$\begingroup$

With the amazing tool fSpy I wanted to reconstruct the interior of a house. I was able to import the fSpy project into Blender, do some basic modelling and UV "Project from view" seemed to work fine.

However, some parts of the image are not correctly UV projected, see image below. The corner of the table in the lower left is not projected to, which is to be expected outside the camera view, but not within camera boundaries! Additionally, when looking at the tray on top of the table, the background image does not align with the UV projected lower portion of the tray.

  • Question 1: In the 3D view (right), why does the UV projection not line up with the preview in the UV editor (left)?
  • Question 2: How to make it work?

Edit 1: I was able to answer the questions myself, see below. After some more trial and error I found an even better solution than described below. I will notes on that later this week!

Edit 2: Title was updated to better reflect what the solution does. Original title was: "How to project a texture on objects in an (fSpy-imported) scene?"

Two images, the UV editor and 3D camera preview. The UV coordinates seem to not be correctly projected in the camera view.

Additional notes

I suspect this UV coordinates that are off boundaries (x<0 or y<0) are somehow being automatically translated somehow, but I don't really know.

I made two small additional changes to better see what's going on. Cutting out those changes did not solve the problem.

  1. In the shader nodes editor, I have set the image texture from Repeat to Clip, so image on the right the problematic area is black and not couch-coloured.
  2. For the table object I added a subdivision modifier set to Simple (not Catmull-Clark). Without it the deformation on the table-top would be based on two triangles with even works deformation.

Related questions

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ @Gorgious Thank you and you're perfectly right! At the same time as you wrote I discovered the reason and the solution myself. See my answer below. $\endgroup$
    – creativecoding
    Commented Mar 1, 2021 at 16:36

3 Answers 3

Reset to default
4
$\begingroup$

I was able to solve the problem after projecting more objects in the scene. The UV coordinates are mapped correctly into the scene from the point of view of a flat surface (like the table top), but because of that not from the point of view of the camera. It does not have anything to do with coordinates being "off bounds".

Imagine a ladder lying on the floor. The rungs have equal distances, but seen "in perspective" the distance between two rungs is getting smaller the further away they are. This non-linearity in perspective causes the linear UV projection on the table top to stretch unintendedly for big faces.

The problem can be solved by having more UV vertex information, i.e. more vertices in the mesh. This way Blender will not stretch the image on a projected face linearly, but follow the non-linearity specified by the additional vertices. Technically this will still stretch the projected image linearly for each face (i.e. not as intended), but this won't be noticable, if the faces are small enough.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Nice explanation! It's been needed for a while, now. There are tutorial videos out there which describe this as a 'bug', and look forward to a 'fix'. $\endgroup$
    – Robin Betts ♦
    Commented Mar 1, 2021 at 17:04
  • 1
    $\begingroup$ @RobinBetts I stand corrected. After playing around a bit with it I understand the limitation : Linearily interpolating UVs is only good if all the vertices are at the same distance to the camera, else the non-linear depth perception will alter the result. I thought I remembered a thread on devtalk where a developer said this was a bug/limitation... May as well never have happened :) When in fact there is nothing that can be done about it. Unless somehow another software does it by default ? Didn't really get the ladder comparison at first. Maybe a diagram would help ? $\endgroup$
    – Gorgious
    Commented Mar 2, 2021 at 19:30
  • 2
    $\begingroup$ As a matter of fact, it doesn't happen when using an Orthographic Camera :) $\endgroup$
    – Gorgious
    Commented Mar 2, 2021 at 19:34
  • 3
    $\begingroup$ Hi, @Gorgious.. is one thing being left out here, render triangulaion? As I understand it, the only coordinates mapped Fustrum-Space to UV-space are those at their corners. Reconstruction of texture between those points is by barycentric interpolation? Which does not correspond to the perspective gradient between the points? But which is approximated better and better, as the size of the triangles is decreased? (Er.. I'll just wedge my head into the corner of the room to stop it spinning) $\endgroup$
    – Robin Betts ♦
    Commented Mar 2, 2021 at 20:17
  • 1
    $\begingroup$ @RobinBetts I have added another answer where I describe the solution I ended up with. $\endgroup$
    – creativecoding
    Commented Mar 6, 2021 at 23:01
3
+1000
$\begingroup$

I experimented a bit further I found that baking of the image texture provides me with exactly the results I was hoping for without subdividing the mesh!

Here is I how got it working:

  • UV-unwrap the object that should be textured.

Choose something like Smart UV Project that keeps all your UV vertices within the image and all faces non-overlapping. Project From View is not necessary! A bit of spacing between islands can make sense.

Keeping all UV vertices within image bounds is important. Otherwise, due to perspective distortion, not all colour information could be stored in the baked image so some parts of the model would be left untextured. Please see other answer for more on that.

  • Create a material to be used for baking.

enter image description here

The trick here is to use the Window output as texture coordinate. Other shaders seem to work, too. You'll see that the object will look funny now, when you rotate the 3D view (material preview).

  • Create a new image.

This image will be baked to. Image creation is possible in the UV/Image Editor. The image size specifies the level of detail of the bake.

  • Select the unconnected image texture node, which is set to the image to be baked to.

If that sounds weird, check the manual why this step is necessary.

  • Set up baking and bake the texture!

Baking currently only works in the Cycles render engine. I limited baking to the Diffuse pass only, without any lighting, but much more is possible here of course.

When baking is done, create a new material with a fairly standard setup as in the screenshot below and select the baked image as image texture. If you compare the 3D views in the lower left on both screenshots, then you'll see that in the screenshot below the tabletop has a even weirder texture. However, that texture now makes sense when viewed in perspective, as in the 3D view on the top right!

You might notice the table top is baked twice. That happens, because camera rays pierce through both faces, but that's okay, because we won't see the things underneath the table anyway.

enter image description here

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ Brilliant. I hope I've got the reason it's working right? BTW, I guess, (belt-and-braces for readers of this answer) you could explicitly plug your new UV Map (node) into the disconnected image-texture while baking, so as not to rely on that map being active. $\endgroup$
    – Robin Betts ♦
    Commented Mar 7, 2021 at 9:20
  • 1
    $\begingroup$ @creativecoding Thanks for the answer :) Finally ... using Window coordinates is common use, but I think all fails (including mine) came from using "Project from View" unwrap, that I just noticed how stupid it was :) "Project from View Bounds" or "Smart Unwrap" works perfectly. $\endgroup$
    – vklidu
    Commented Mar 7, 2021 at 15:39
  • $\begingroup$ @vklidu Thank you! Side note: Project from View Bounds will result in overlapping faces, when you have a "closed" mesh. Overlapping faces confuse Blender when baking, leading to artifacts in the baked image. $\endgroup$
    – creativecoding
    Commented Mar 7, 2021 at 23:58
  • 1
    $\begingroup$ @creativecoding Hm... I had no issue with that and technically it shouldn't be a confusion for baking since overlapping faces share exactly the same pixels at that place. Will check that again :) $\endgroup$
    – vklidu
    Commented Mar 8, 2021 at 7:21
  • 1
    $\begingroup$ Benefit of Project-from-View is efficient use of original pixels on texture in term of depth - hires for close / lowers for far parts. This is not happening with Smart Unwrap or others, and you have to care if UV of close up faces are not smaller than original, that can easily became blurry. Sure for reconstructions or more camera movement where you need all faces (not-overlapped) to recreate texture - Project from View is useless. $\endgroup$
    – vklidu
    Commented Mar 9, 2021 at 16:29
0
$\begingroup$

to project the texture, exactly from the image, you will want to do (in edit mode) U>project from view and then apply the reference as a material

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for the response! However, I think what you suggested is what I tried first. This resulted in distorted mapping of image information from the UV space to the XYZ space. Please see the (chronologically) first answer to the (supposed) cause for this problem. $\endgroup$
    – creativecoding
    Commented Mar 8, 2021 at 0:07
  • $\begingroup$ hmmm... I see what you're saying, is the table the focal point (if I'm using that phrase correctly) of the image? if not, you could project from view, then move the table and its carpet to the side to hide the distortion. $\endgroup$
    – EverythingLeafwing
    Commented Mar 8, 2021 at 15:28
  • $\begingroup$ The table is not in the focal point or in a vanishing point. But the idea in the beginning was to rebuild the scene in Blender as it is seen on the background image, so moving the problematic things out of the way is not an option unfortunately. :-) $\endgroup$
    – creativecoding
    Commented Mar 9, 2021 at 9:01
  • $\begingroup$ try remodelling it, or subdeviding it? maybe mark random seams to sea if that changes anything? I'm honestly confused as too what's happening to the uv's too $\endgroup$
    – EverythingLeafwing
    Commented Mar 9, 2021 at 15:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .