1
$\begingroup$

I've just about got to grips with the UV Project modifier. I've unwrapped my object into a UV map with the image I want to project, and then applied it using a camera as a projector.

However, I want to project from multiple angles. So I created another UV map the same way with my second image and created another UV Project modifier. But this won't work, because I can only select 1 UV map at a time. I want the two projections to be blended together (or, even better, chosen programmatically according to which surface normal has the closer angle to the projector when I eventually move to automating this process in Python).


Here's what it looks like when projecting from the camera on the left. It's projecting the same image it would see if the house were textured properly. Likewise, the camera on the right has an image of the properly textured house from its angle, ready to "fill in the gaps" of what the left camera cannot see.

One of two image projections applied to the object

P.S. Please try not to include steps that could only be done manually/by eye. Because I am applying research to a large collection of objects, this will have to be done programmatically eventually.

$\endgroup$
  • 2
    $\begingroup$ Related blender.stackexchange.com/questions/75837/… $\endgroup$ – lemon Mar 31 '17 at 7:13
  • $\begingroup$ I tried stacking the two projection images into a single image and then used multiple projectors in the modifier like in the related question, but because the images and projectors have perspective, I can't just move the projectors until the geometry is aligned. I need to somehow keep the projectors in place while adding an offset to the image they project. $\endgroup$ – CJxD Apr 1 '17 at 15:28
  • $\begingroup$ I've just gave a look to your file. And I'd like to be sure to understand the question, finally: you wanted to use UV project in order to reconstitute the UV maps from the already rendered images (as a test of the method)? $\endgroup$ – lemon Apr 4 '17 at 11:45
  • $\begingroup$ You can calculate a perspective projection and define the UV maps from that, but some UV faces will overlap $\endgroup$ – lemon Apr 5 '17 at 4:41
  • $\begingroup$ @lemon Could you elaborate further on this? Imagine this use case: I have a Blender model of a real-life object but it's untextured. I also have multiple images of the object taken with a real-life camera that I know the parameters of. What's the best way to project these images to recreate as much of the object's look as possible? $\endgroup$ – CJxD Apr 5 '17 at 9:26
3
$\begingroup$

Following the last comments, this is more some guidelines than a real answer.

What you can do/code inside Blender is an addon based on the same principle as UV project modifier. The principles are the following:

  • Given a camera and its parameters, you can obtain its projection matrix. For instance have a look at this.

  • You can also use/create an UV map (to be built by the code). In this addon, I image that you could parameter one UV map by camera (that may be needed or not, depending on the way you'll define the materials/textures correspondences).

  • Now you can loop over your model polygons (faces), test their normals against the concerned cameras axis and so determinate which polygons are best concerned by which camera.

  • Knowing each polygon/camera pairs, and the previous projection matrix, you can calculate the UVs of the polygons, then associate them to the corresponding UV map.

You are done...

Eventually you can also dynamically test camera's parameters changes (tuned by user to adjust the uv/image mapping) and auto update the UV map (performance to be taken into account).

That should be relatively simple.

But the problem is you will have partially hidden polygons due to the projection (it is true too for orthographic projection) and these polygons will overlap on the UV map.

edit:

A more complete answer in an addon that calculates the UV projections of the mesh, depending of the camera positions.

As indicated above, it uses the same principle as the UV project modifier (in the UI part).

enter image description here

(the blank parts in the gif are parts where no image is given).

For each camera, you need to indicate the corresponding UV map and material slot to use. Then the script calculates the polygons oriented to each camera and for each polygon the projected coordinates to be assigned to the UV map.

The code below is commented for more information:

import bpy
import time
from mathutils import Vector
from operator import itemgetter

#Used to store information related to each camera setting
class UVPPCameraSetting:
    def __init__( self, scene, camera, obj, uvMap, materialSlotIndex ):
        self.camera = camera
        self.uvMap = uvMap
        self.materialSlotIndex = materialSlotIndex
        #Z axis of the camera translate in world (Z axis is opposite to view)
        self.zInWorld = Vector( (0, 0, 1) )
        self.zInWorld.rotate( self.camera.matrix_world.to_euler() )
        #To keep polygons to handle for this setting
        self.polygons = set()
        #Camera parameters
        #Matrix to convert from object coordinate to camera coordinates
        self.toCameraMatrix = camera.matrix_world.inverted() * obj.matrix_world
        #The frame is composed of the coordinates in the camera view
        frame = [v / v.z for v in camera.data.view_frame(scene=scene)]
        #Get the X, Y corners
        self.minX = min( v.x for v in frame )
        self.maxX = max( v.x for v in frame )
        self.minY = min( v.y for v in frame )
        self.maxY = max( v.y for v in frame )
        #Precalculations to avoid to repeat them when applied to the model
        self.deltaX = self.maxX - self.minX
        self.deltaY = self.maxY - self.minY
        self.offsetX = self.minX / self.deltaX
        self.offsetY = self.minY / self.deltaY

    #Calculate the UV coordinateds from the object coordinates
    def CalcUV( self, objCo ):
        #Object in camera view
        camCo = self.toCameraMatrix * objCo
        #Z is "inverted" as camera view is pointing to -Z of the camera
        z = -camCo.z
        try:
            #Translates x and y to UV coordinates
            x = (camCo.x / (self.deltaX * z)) - self.offsetX        
            y = (camCo.y / (self.deltaY * z)) - self.offsetY        
            return x, y, z
        except:
            #In case Z is zero
            return 0.5, 0.5, 0

def UVPerspectiveProject( scene, obj, camSettingList ):
    print( '------------------------' )
    startTime = time.time()

    matrix_world = obj.matrix_world
    rotation_world = matrix_world.to_euler()

    #Assign polygon to its corresponding camera considering faces most aligned to camera view
    for p in obj.data.polygons:
        normalInWorld = Vector( p.normal )
        normalInWorld.rotate( rotation_world )
        camSetting, maxDot = max( ((c, normalInWorld.dot( c.zInWorld)) for c in camSettingList), key=itemgetter(1) )
        camSetting.polygons.add( p.index )

    print( 'precalculations. elapse in seconds', time.time() - startTime )

    loops = obj.data.loops
    vertices = obj.data.vertices
    #For each setting, 
    for cs in camSettingList:
        camera = cs.camera
        uvMap = cs.uvMap
        polygons = cs.polygons
        projectedCo = {} #Storage to avoid multiple calculations of the same world_to_camera_view
        #Go through all polygons
        for p in obj.data.polygons:
            #If the polygon corresponds to the setting
            if p.index in polygons:
                #Assign the material index
                p.material_index = cs.materialSlotIndex
                #Calculate each vertex uv projection
                for i, vi in [(i, loops[i].vertex_index) for i in p.loop_indices]:
                    if vi not in projectedCo: #not already calculated for this cam
                        x, y, z = cs.CalcUV( vertices[vi].co )
                        projectedCo[vi] = (x,y)
                    uvMap.data[i].uv = projectedCo[vi]
            #If not, uv are set to (0,0)
            else:
                for i in p.loop_indices:
                    uvMap.data[i].uv = (0, 0)

    print( 'done. elapse in seconds', time.time() - startTime )

def TargetObjExists( context ):
    try:
        scn = context.scene
        obj = scn.objects[scn.uvPerspectiveProject.object_name]
        return obj.type == 'MESH'
    except:
        pass
    return False

def IsCamera( context, camera_name ):
    try:
        camera = context.scene.objects[camera_name]
        return camera.type == 'CAMERA'
    except:
        pass
    return False

def SettingsAreOK( context ):
    try:
        scn = context.scene
        settings = scn.uvPerspectiveProject
        obj = scn.objects[settings.object_name]
        assert obj.type == 'MESH'

        for item in settings.cameras_settings:
            camera = scn.objects[item.camera_name]
            assert camera.type == 'CAMERA'
            uv_map = obj.data.uv_layers[item.uv_map_name]
            materialSlot = obj.material_slots[item.material_slot_name]

        return True
    except:
        pass
    return False

class UVPerspectiveProjectOperator( bpy.types.Operator ):
    bl_idname = "lemon.uvperspectiveprojectoperator"
    bl_label = "UV perspective project"
    bl_options = {'REGISTER'}

    @classmethod
    def poll(self, context):
        return context.mode == 'OBJECT' and SettingsAreOK( context )

    def invoke(self, context, event):
        scn = context.scene
        settings = scn.uvPerspectiveProject

        #try:
        obj = scn.objects[settings.object_name]
        cameraSettings = [UVPPCameraSetting( scn, scn.objects[item.camera_name], obj, obj.data.uv_layers[item.uv_map_name], obj.material_slots.find( item.material_slot_name ) ) for item in settings.cameras_settings]
        UVPerspectiveProject( scn, obj, cameraSettings )
        #except:
            #pass

        return { 'FINISHED' }

class UVPerspectiveProjectCamSettingsActions( bpy.types.Operator ):
    bl_idname = "lemon.uvperspectiveprojectpanelcamlistactions"
    bl_label = ""

    action = bpy.props.EnumProperty( items = ( ('UP', "Up", ""), ('DOWN', "Down", ""), ('REMOVE', "Remove", ""), ('ADD', "Add", ""), ) )

    def invoke(self, context, event):

        scn = context.scene
        settings = scn.uvPerspectiveProject
        index = settings.camera_setting_index

        if self.action == 'DOWN' and index < len(settings.cameras_settings) - 1:
            settings.cameras_settings.move( index, index + 1 )
            settings.camera_setting_index += 1

        elif self.action == 'UP' and index > 0:
            settings.cameras_settings.move( index, index - 1 )
            settings.camera_setting_index -= 1

        elif self.action == 'REMOVE':
            settings.cameras_settings.remove( settings.camera_setting_index )
            settings.camera_setting_index -= 1

        elif self.action == 'ADD':
            item = settings.cameras_settings.add()
            settings.camera_setting_index = len(settings.cameras_settings) - 1

        return {"FINISHED"}

class UVPerspectiveProjectCamSettingsPanel( bpy.types.UIList ):

    def draw_item(self, context, layout, data, item, icon, active_data, active_propname, index):

        scn = context.scene
        settings = scn.uvPerspectiveProject

        try:
            obj = scn.objects[settings.object_name]
        except:
            pass

        suffix = " (" + str( index + 1 ) + ")"

        row = layout.row()
        col = row.column( align = True )
        col.prop_search( item, "camera_name", scn, "objects", text="Camera" + suffix  )
        col.prop_search( item, "uv_map_name", obj.data, "uv_layers", text="UV map" + suffix )
        col.prop_search( item, "material_slot_name", obj, "material_slots", text="Material" + suffix )

    def invoke(self, context, event):
        pass   

class UVPerspectiveProjectPanel( bpy.types.Panel ):
    bl_idname = 'lemon.uvperspectiveprojectpanel'
    bl_space_type = "VIEW_3D"
    bl_region_type = "UI"
    bl_label = "UV perspective project"

    @classmethod
    def poll(self, context):
        return context.mode == 'OBJECT'

    def draw(self, context):
        layout = self.layout
        scn = context.scene

        settings = scn.uvPerspectiveProject

        layout.row().prop_search( settings, "object_name", scn, "objects", text="Object" )

        if TargetObjExists( context ):
            layout.row().label( text="Cameras settings:" )

            row = layout.row()        
            row.template_list( "UVPerspectiveProjectCamSettingsPanel", "", settings, "cameras_settings", settings, "camera_setting_index", rows = 3 )

            col = row.column( align = True )
            col.operator( UVPerspectiveProjectCamSettingsActions.bl_idname, icon='ZOOMIN', text="" ).action = 'ADD'
            col.operator( UVPerspectiveProjectCamSettingsActions.bl_idname, icon='ZOOMOUT', text="" ).action = 'REMOVE'
            col.separator()
            col.operator( UVPerspectiveProjectCamSettingsActions.bl_idname, icon='TRIA_UP', text="" ).action = 'UP'
            col.operator( UVPerspectiveProjectCamSettingsActions.bl_idname, icon='TRIA_DOWN', text="" ).action = 'DOWN'

            layout.row().operator( UVPerspectiveProjectOperator.bl_idname, text="Apply" )


class UVPerspectiveProjectCamSettingsProps( bpy.types.PropertyGroup ):
    camera_name = bpy.props.StringProperty()
    uv_map_name = bpy.props.StringProperty()
    material_slot_name = bpy.props.StringProperty()

class UVPerspectiveProjectProps( bpy.types.PropertyGroup ):
    object_name = bpy.props.StringProperty()
    cameras_settings = bpy.props.CollectionProperty( type = UVPerspectiveProjectCamSettingsProps )
    camera_setting_index = bpy.props.IntProperty()

def register():
    bpy.utils.register_module(__name__)
    bpy.types.Scene.uvPerspectiveProject = bpy.props.PointerProperty( type = UVPerspectiveProjectProps )

def unregister():
    del bpy.types.Scene.uvPerspectiveProject
    bpy.utils.unregister_module(__name__)

if __name__ == "__main__":
    register()

$\endgroup$
  • $\begingroup$ Thanks again for this script, it's just what I'm looking for. I'm encountering an issue though: in your .blend file, all of the projections work together, but when I tried to replicate what you did, I can only see one UV Map at a time. All other areas are blank until I select a different UV map. $\endgroup$ – CJxD May 1 '17 at 11:53
  • $\begingroup$ And I don't suppose there's a way to use Blender Render rather than Cycles? I have hundreds of thousands of renders to make. $\endgroup$ – CJxD May 1 '17 at 11:54
  • $\begingroup$ @CJxD, I think that can work for Blender Internal too. Concerning the projections: each UV map will have all projection (this is the way Blender is working) but unneeded faces are set to 0,0,0 (by choice/convention in the script). The principle globally (in both renderer): a face is assigned to a material and a material uses a UV map (via a texture/image in both renderer). $\endgroup$ – lemon May 1 '17 at 12:19
  • $\begingroup$ I get you, but how then were you able to show both projections at the same time if half the faces are set to 0,0,0 for any particular UV map? $\endgroup$ – CJxD May 1 '17 at 13:40
  • $\begingroup$ @CJxD, you can show both projection in 3D view due to the material setting (which can be done also for BI), because faces are assigned to the corresponding material (and that's why the UI of the script links cam/mat/uvmap). But you can't show multiple projections in the image/uv editor. UV set to (0,0) are set so for the non useful faces for this particular UV map. $\endgroup$ – lemon May 1 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.