Why I need to run my script twice to get it to work?

Question

Basically I am trying to write a script to

1. create a square image
2. add some objects to it
3. get the positions of the objects in frame coordinates of the objects.

There is a second part of the script in which I render the frame and also save the positions of the objects in a pickle file but I am not including it to make the script shorter.

The idea is that then I am going to create some bounding boxes to the objects in python and use them for other purposes. So I need to be able to render the image and export object locations in frame coordinates.

If I open the script from blender, the first time I run it it has an error, but the second time I run it it works fine and I get these values

**cube
[0.0035274855948945524, 0.24358138840914842, 0.7366971959744029, 0.9738736891060572]
**********************
**sphere
[0.43452719267736106, 0.5654728073226389, 0.43452716000590824, 0.5654728399940917]
**********************

which are the correct one.

If I run the script from the command line it fails because view3d_camera_border is not working as expected and I get the same error as when I run the script from the GUI:

blender -b --python test_camera_blender.py

So it looks that I have some missing/wrong initial config in the file that after I run it two times it is fixed. I have been tying to debug the whole thing but I am not able to make it work. Any ideas which configuration I am missing?

Script:

"""
Some usefuleful referenes:
- https://blender.stackexchange.com/questions/55300/how-can-i-get-the-x-y-location-of-an-object-at-each-frame-of-animation
- https://blender.stackexchange.com/questions/237428/get-pixel-coords-for-vertex-in-viewport
- https://gist.github.com/simonecesano/2cefcfac7485939a18249c4672e4a5ff
- https://blender.stackexchange.com/questions/6377/coordinates-of-corners-of-camera-view-border
"""

import bpy
import numpy as np
from mathutils import Vector
import bmesh
import bpy_extras

# make the output dimensions square 1920x1920
create_objects = True
im_size = 1920


def to_rad(deg):
    return deg * np.pi / 180


def delete_all_objects():
    """
    Deletes all objects in the current scene
    """
    deleteListObjects = [
        "MESH",
        "CURVE",
        "SURFACE",
        "META",
        "FONT",
        "HAIR",
        "POINTCLOUD",
        "VOLUME",
        "GPENCIL",
        "ARMATURE",
        "LATTICE",
        "EMPTY",
        "LIGHT",
        "LIGHT_PROBE",
        "CAMERA",
        "SPEAKER",
    ]

    # Select all objects in the scene to be deleted:

    for o in bpy.context.scene.objects:
        for i in deleteListObjects:
            if o.type == i:
                o.select_set(False)
            else:
                o.select_set(True)
    # Deletes all selected objects in the scene:

    bpy.ops.object.delete()


def toggle_camera(context):
    for area in context.screen.areas:
        if area.type == "VIEW_3D":
            space = area.spaces.active
            r3d = space.region_3d
            plane_no = r3d.view_rotation @ Vector((0, 0, -1))
            region = area.regions[-1]
            area.spaces[0].region_3d.view_perspective = 'CAMERA'
            area.spaces[0].region_3d.view_camera_offset = (0,0)
            area.spaces[0].region_3d.view_camera_zoom = -1000
            break
    else:
        assert False, "Requires a $D view"
    return region, r3d, area


def region_3d(context):
    for area in context.screen.areas:
        if area.type == "VIEW_3D":
            space = area.spaces.active
            r3d = space.region_3d
            plane_no = r3d.view_rotation @ Vector((0, 0, -1))
            region = area.regions[-1]
            break
    else:
        assert False, "Requires a $D view"
    return region, r3d, area


def vertex_positions_2d(obj, region, r3d):
    bm = bmesh.new()
    bm.from_mesh(obj.data)
    bm.transform(obj.matrix_world)

    v_2d_co = np.array(
        [
            bpy_extras.view3d_utils.location_3d_to_region_2d(region, r3d, v.co)
            for v in bm.verts
        ]
    )
    return v_2d_co


def view3d_camera_border(scene, region, r3d):
    obj = scene.camera
    cam = obj.data
    frame = cam.view_frame(scene=scene)
    print("**cam frame")
    for f in frame:
        print(f)
    # move into object space
    frame = [obj.matrix_world @ v for v in frame]
    print("**cam frame world debug")
    for f in frame:
        print(f)
    # move into pixel space
    frame_px = [
        bpy_extras.view3d_utils.location_3d_to_region_2d(region, r3d, v, default='NOOTTT') for v in frame
    ]
    print("**frame_px debug")
    for f in frame_px:
        print(f)
    return frame_px


def rescale_2d_to_cam(xy, cbounds):
    x_min, x_max, y_min, y_max = cbounds
    x, y = xy
    x = (x - x_min) / (x_max - x_min)
    y = (y - y_min) / (y_max - y_min)
    return list((x, y))


def get_min_max_from_list(l):
    min_x = np.min([v[0] for v in l])
    max_x = np.max([v[0] for v in l])
    min_y = np.min([v[1] for v in l])
    max_y = np.max([v[1] for v in l])
    return [min_x, max_x, min_y, max_y]


def to_camera_view(scene, cam, obj):
    _c = []
    for vert in obj.data.vertices: 
        # local to global coordinates
        co = obj.matrix_world @ vert.co
        coords2d = bpy_extras.object_utils.world_to_camera_view(scene, cam, co)
        # print("coords2d: {},{}".format(coords2d.x, coords2d.y))
        _c.append(coords2d)
    return _c


def create_objects_func():
    # add cube
    bpy.ops.mesh.primitive_cube_add(location=(2, -3.9, 7.2), rotation=(0, to_rad(45), 0))
    cube = bpy.data.objects["Cube"]
    cube.name = "cube"

    # Make a sphere at the origin named 'sphere'
    bpy.ops.mesh.primitive_uv_sphere_add(location=(0, 0, 0), rotation=(0, 0, 0))
    sphere = bpy.data.objects["Sphere"]
    sphere.name = "sphere"

    ## Add a camera facing the cube from 45 top angle
    bpy.ops.object.camera_add(
        location=(15, 0, 15), rotation=(to_rad(45), 0, to_rad(90))
    )
    camera = bpy.data.objects["Camera"]
    camera.name = "camera"

    ## Add a light source
    bpy.ops.object.light_add(
        type="SUN", location=(25, 0, 25), rotation=(0, to_rad(28), to_rad(20))
    )
    lamp = bpy.data.objects["Sun"]
    lamp.name = "sun"
    lamp.data.energy = 1


def main():
    for scene in bpy.data.scenes:
        print(scene)
        scene.render.resolution_x = im_size
        scene.render.resolution_y = im_size

    context = bpy.context
    scene = context.scene
    toggle_camera(context)

    print("______________________")
    print('**debug resolution', scene.render.resolution_x, scene.render.resolution_y)
    

    region, r3d, area = region_3d(context)
    
    
    if create_objects:
        delete_all_objects()
        create_objects_func()
        scene.camera = bpy.data.objects["camera"]

    area.spaces[0].region_3d.view_perspective = 'CAMERA'
    area.spaces[0].region_3d.view_camera_offset = (0,0)
    area.spaces[0].region_3d.view_camera_zoom = -1000

    # get the cube position in world coordinates
    cube = bpy.data.objects["cube"]
    loc, rot, scale = cube.matrix_world.decompose()
    print("**cube world coords")
    print("**loc", loc)

    # Get info of objects
    print("**DEBUG", region, r3d)
    frame_px = view3d_camera_border(scene, region, r3d)
    print("**frame_px", frame_px)

    cam_min_x = np.min([v[0] for v in frame_px])
    cam_max_x = np.max([v[0] for v in frame_px])
    cam_min_y = np.min([v[1] for v in frame_px])
    cam_max_y = np.max([v[1] for v in frame_px])
    cam_bounds = [cam_min_x, cam_max_x, cam_min_y, cam_max_y]
    print("**camera bounds", cam_bounds)

    print(region, r3d)
    final_positions = {}

    for name in ['cube', 'sphere']:
        
        positions = vertex_positions_2d(bpy.data.objects[name], region, r3d)
        positions_scaled = [rescale_2d_to_cam(xy, cam_bounds) for xy in positions]
        print(f"**{name} ")
        print(get_min_max_from_list(positions_scaled))
        final_positions[name] = get_min_max_from_list(positions_scaled)
        
        
        print("**********************")
    
    print("______________________")


main()

```

Answer 1

The error does not pop up for the first time only, you can provoke it also when running the script multiple times. After running the script, move into the sphere (such that you are inside) and run the script again. This will tigger the error again.

The reason is that, as you noticed, view3d_camera_border is not working correctly, but actually it does. Within that method you're calling

bpy_extras.view3d_utils.location_3d_to_region_2d(region, r3d, v, default='NOOTTT') for v in frame

and when inspecting the debug prints you should notice there are NOOTTT values in frame_px causing the error:

... ufunc 'minimum' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> None

As the docs state for location_3d_to_region_2d:

default – Return this value if coord is behind the origin of a perspective view.

This gives you the clue that something with the camera view from the 3D region is wrong. If you run the script for the second time it works because the scene camera's view and the region's view do match accidently. And thats why you can provoke the error again if you're moving the region's camera to a position where "coord is behind the origin of a perspective view".

What you need to do is to update the region after you changed the the view data so you get the correct camera data to work with. You have two places in code where you do:

area.spaces[0].region_3d.view_perspective = 'CAMERA'
area.spaces[0].region_3d.view_camera_offset = (0,0)
area.spaces[0].region_3d.view_camera_zoom = -1000

one in def toggle_camera and one in def main (you should use toggle_camera there too). Change that to:

area.spaces[0].region_3d.view_perspective = 'CAMERA'
area.spaces[0].region_3d.view_camera_offset = (0,0)
area.spaces[0].region_3d.view_camera_zoom = -1000
area.spaces[0].region_3d.update()    # <-- here

and it works like desired.