2
$\begingroup$

I'm running a script in Blender 2.79b to save 2D proyections of a 3D face model with annotated facial landmarks, I apply random transformations to the mesh and the landmarks, render the mesh and save a file with the proyected landmarks in image coordinates.

I've tested this in the console and it runs fine, but the function ray_cast behaves differently if I run the script (I've tried running it from the Text Editor, with exec from the Python Console and from the linux terminal) rather than writing it line by line in the Blender Python Console.

Here is the code

from bpy_extras.object_utils import world_to_camera_view
import bpy
from mathutils import Matrix, Vector

import math
import numpy as np
import csv
import os
import time

print(os.getcwd())

MODELS_PATH = "models"
IMAGES_PATH = "images"
MODEL_ITERS = 1
IMAGE_SIZE = 500

np.random.seed(0)

class Mod:
    def __init__(self,row):
        self.name = row[0]
        self.values = row[1:]

    def iterValues(self):
        for i in range(len(self.values)//3):
            yield Vector(tuple(map(float,self.values[i*3:i*3+3])))

    def getFileName(self):
        return os.path.join(MODELS_PATH,"{}.obj".format(self.name))

    def getNumberedOutPath(self,i):
        return os.path.join(IMAGES_PATH,"{}_{}.png".format(self.name,i)),os.path.join(IMAGES_PATH,"{}_{}.pts".format(self.name,i))

## Set Environment
bpy.data.worlds['World'].light_settings.use_environment_light = True
bpy.data.worlds['World'].light_settings.environment_energy = 1

## Set camera
cam_c = bpy.data.cameras['Camera']
cam_o = bpy.data.objects['Camera']
# Location 0,0,3
cam_o.location = (0,0,3)
# Rotation 0,0,0
cam_o.rotation_euler = (0,0,0)
# Focal Length -> 85
cam_c.lens = 85
scene = bpy.data.scenes['Scene']
bpy.data.scenes['Scene'].render.resolution_x = IMAGE_SIZE
bpy.data.scenes['Scene'].render.resolution_y = IMAGE_SIZE
bpy.data.scenes['Scene'].render.resolution_percentage = 100

## Define lighting
lamp_o = bpy.data.objects['Lamp']
lamp_l = bpy.data.lamps['Lamp']

# Load models
with open('mod_coords.csv') as ifile:
    reader = csv.reader(ifile)
    models = [Mod(row) for row in reader]

# For each model
for model in models:
    m_path = model.getFileName()

    ## Load object
    bpy.ops.import_scene.obj(filepath=m_path)
    obj = bpy.context.selected_objects[0]

    obj.location = (0, 0, 0)
    obj.rotation_euler = (0, 0, 0)

    for m in obj.material_slots:
        m.material.specular_intensity = 0

    # Normalize object
    vts_np = np.array([e.co for e in obj.data.vertices])
    desp = vts_np.mean(axis=0)
    sca = vts_np.max(axis=0) - vts_np.min(axis=0)
    dx, dy, dz = desp
    s = 1 / sca[1]
    M1 = Matrix(((s, 0, 0, -s * dx), (0, s, 0, -s * dy), (0, 0, s, -s * dz), (0, 0, 0, 1)))

    obj.data.transform(M1)

    # For each random example
    for i in range(MODEL_ITERS):
        rx = np.random.uniform(-0.610865, 0.610865)
        ry = np.random.uniform(-1.5708, 1.5708)
        rz = np.random.uniform(-0.523599, 0.523599)
        Rx = Matrix.Rotation(rx, 4, 'X')
        Ry = Matrix.Rotation(ry, 4, 'Y')
        Rz = Matrix.Rotation(rz, 4, 'Z')
        # Apply rotation
        R = Rx * Ry * Rz
        obj.data.transform(R)
        obj.data.update()

        Mf = R * M1
        ## Get points
        points_list = []
        for p in model.iterValues():
            p = Vector(p)
            if math.isnan(p.x):
                points_list.append([-1,-1])
            else:
                tp = Mf * p

                # position
                x, y, z = world_to_camera_view(scene, cam_o, tp) * IMAGE_SIZE
                y = IMAGE_SIZE - y
                # visibility
                o = cam_o.location
                d = tp - o

                r_bool, r_loc, _, _, r_obj, _ = scene.ray_cast(o, d)
                print(o,d,r_bool,r_loc)
                # check if r_loc y tp are equal within error
                error = 1e-1
                visib = (r_loc - tp).magnitude < error
                if not visib:
                    print(r_loc,tp)
                points_list.append([x,y] if visib else [-2,-2])

        ## Random Lighting
        rx = np.random.uniform(-5, 5)
        ry = np.random.uniform(-5, 5)
        r_energy = np.random.uniform(0.1, 2)
        r_shadow = np.random.uniform(0.1, 10)
        lamp_o.location = (rx, ry, 2.5)
        # TODO apply energy and shadow

        ## Save image and points
        im_path, pts_path = model.getNumberedOutPath(i)
        # Render image
        bpy.data.scenes['Scene'].render.filepath = im_path
        bpy.ops.render.render(write_still=True)
        # Save points
        with open(pts_path,'w') as ofile:
            ofile.writelines([
                "version: 1\n",
                "n_points: {}\n".format(len(points_list))] +
            ["{} {}\n".format(x[0],x[1]) for x in points_list] +
            ["}\n"]
            )

        ## Undo random rotation
#        R.invert()
#        obj.data.transform(R)

#    bpy.ops.object.select_all(action='DESELECT')
#    obj.select = True
#    bpy.ops.object.delete()
    break

Here is an example of the proyected mesh

Proyected face mesh

But every single call to ray_cast returns False. If I comment out the last 5 code lines and add a break after the first model loop I get to keep the model loaded in the 3D View with the applied transformation (this way I get some debugging capabilities :D). As I'm printing the origin and destination variables passed to ray_cast I can call again that function from the Python Console, and the result differs from the one obtained running the script, ray_cast returns True and the colission coordinates matches the target point if there is no occlusion or another point otherwise (This is the desired functionality).

If I rerun the script keeping the first Object and overlaying the second one (as the random seed is fixed), the ray_cast function works as expected, but the render produces some artifacts due to both meshes overlaying. My first hypothesis was Blender had no time enough to apply the transformations so I called time.sleep(5) after loading and transforming the model, but the result was the same.

I have no clue about what to do. Has anyone came by a similar situation? Am I doing something wrong?

Thanks a lot!

$\endgroup$
1
$\begingroup$

I couldn't figure out the reason for the error, but I could overcome it by calling ray_cast model's method instead of the scene's method.

r_bool, r_loc, _, _, r_obj, _ = scene.ray_cast(o, d)

becomes

r_bool, r_loc, _, _ =  obj.ray_cast(o, d)

As I already knew the object I wanted the ray to hit.

I hope this might help somebody.

$\endgroup$
  • $\begingroup$ Object ray cast uses object space, (where (0, 0, 0) is the object origin) whereas Scene raycast (AFAIK) uses global space, where (0, 0, 0) is the scene or global origin.. o and d, the origin and direction of the ray, in examples above would need to be adjusted accordingly for objects with non identity matrix worlds. $\endgroup$ – batFINGER May 10 at 10:26
  • $\begingroup$ Thank you for your comment. In this case, both object and scene share the (0,0,0) origin, so I don't think that should be the issue. $\endgroup$ – user3325504 May 11 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.