Low level vs High level object creation speed UPDATE: Fast Data-Block Link to scene

Question

I am trying to make a lot of objects with a python script. I have read multiple posts about low level vs high level programming and the slow-down due to scene-update of all objects after the creation of every one, and have based my second loop on them ('low level' programming).

The problem is that while the low level loop runs much faster, especially with the more objects added (I've tried up to 26,000), the scene never updates after creation. The windows 'waiting wheel' just spins and spins.

First, here is my data. I've shortened it to just be the first 1,000 rows. The view here is as a Pandas DataFrame, but I've converted it to a numpy array for speed before use.

        id  xcentre  ycentre  zcentre  xlength  ylength  zlength
0   783782  10236.5  14098.0    596.5      3.0      4.0      1.0
1   783783  10245.5  14091.0    596.5      5.0      2.0      1.0
2   783784  10247.5  14091.5    595.0      1.0      1.0      2.0
3   783785  10248.0  14093.0    594.5      4.0      2.0      1.0
4   783786  10245.5  14093.5    594.5      1.0      1.0      1.0
5   783787  10248.5  14087.5    595.5      1.0      1.0      1.0
6   783817  10240.5  14092.5    596.5      1.0      1.0      1.0
7   783818  10232.5  14099.5    596.5      1.0      1.0      1.0
8   783819  10234.5  14097.5    596.5      1.0      1.0      1.0
9   783820  10237.0  14095.5    596.5      2.0      1.0      1.0
10  783821  10238.5  14095.5    596.5      1.0      3.0      1.0
11  783822  10244.5  14089.5    596.5      1.0      1.0      1.0
12  783823  10246.5  14087.5    596.5      1.0      1.0      1.0
13  783833  10245.5  14091.5    595.5      3.0      1.0      1.0
14  783834  10234.0  14099.0    596.5      2.0      2.0      1.0
15  783835  10242.0  14092.0    596.5      2.0      2.0      1.0
16  783836  10246.0  14089.0    596.5      2.0      2.0      1.0
17  783837  10249.5  14082.0    596.5      1.0      4.0      1.0
18  783838  10247.0  14090.5    595.5      2.0      1.0      1.0
19  783839  10246.5  14096.0    596.0      7.0      8.0      2.0

Here is the first script ('high level'):

#shorten the DataFrame and convert to numpy matrix
cut1 = cut[0:1000]
cut1 = cut1.as_matrix() 

#Make the material for the cubes
purple = bpy.data.materials.new("Purple")
purple.diffuse_color = (1.0, 0, 1.0)

#Make the material for the labels
label = bpy.data.materials.new("LABEL")
label.diffuse_color = (1.0,1.0,1.0)

#Make the label and parent
bpy.ops.object.text_add(view_align=False, location=(cut1[:,1].mean(), cut1[:,2].mean(),cut1[:,3].max()+500), rotation=(0,4.72,0))
txt = bpy.context.object
txt.name = "Cubes"
txt.data.body = "Cubes"
txt.dimensions = 200, 100, 0
txt.data.materials.append(label)

start = time.time()
for i in range(len(cut1)):

    bpy.ops.mesh.primitive_cube_add(location=(cut1[i,1], cut1[i,2], cut1[i,3]))
    block = bpy.context.object
    block.scale = (cut1[i,4]/2, cut1[i,5]/2, cut1[i,6]/2)
    block.data.materials.append(purple)
    block.parent = txt
    block.matrix_parent_inverse = txt.matrix_world.inverted()



end = time.time()

print("elapsed time is: {}".format(start-end))  

This runs in 12 seconds, and the results are perfect, where the blocks are exactly as I want. The scene interacts (zoom, rotate, etc) very fast.

The problem here is that after about 3000 objects, the script slows down dramatically. By 10,000 objects it takes well over a second per object. To make the 26,000 objects that I want would take days.

Here is the second script ('low level'):

#shorten the DataFrame and convert to numpy matrix
cut1 = cut[0:1000]
cut1 = cut1.as_matrix() 

#Make the material for the cubes
purple = bpy.data.materials.new("Purple")
purple.diffuse_color = (1.0, 0, 1.0)
#Make the material for the labels
label = bpy.data.materials.new("LABEL")
label.diffuse_color = (1.0,1.0,1.0)

#Make the label and parent
bpy.ops.object.text_add(view_align=False, location=(cut1[:,1].mean(), cut1[:,2].mean(),cut1[:,3].max()+500), rotation=(0,4.72,0))
txt = bpy.context.object
txt.name = "Cubes"
txt.data.body = "Cubes"
txt.dimensions = 200, 100, 0
txt.data.materials.append(label)

#make the first cube to be the base for the obs array
bpy.ops.mesh.primitive_cube_add(location=(0,0,0))
ob = bpy.context.object
obs = []
sce = bpy.context.scene
start = time.time()

for i in range(len(cut1)):

    copy = ob.copy()
    copy.location += Vector((cut1[i,1], cut1[i,2], cut1[i,3]))
    copy.scale += Vector((cut1[i,4]/2, cut1[i,5]/2, cut1[i,6]/2))
    copy.data.materials.append(purple)
    copy.data = copy.data.copy()
    copy.hide=True    
    copy.parent = txt
    copy.matrix_parent_inverse = txt.matrix_world.inverted()    
    obs.append(copy)


for ob in obs:
    sce.objects.link(ob)

end = time.time()
print('elapsed time is: {}'.format(end-start))  

sce.update() # don't place this in either of the above loops!    

end2 = time.time()
print("scene update took: {}".format(end2-end))

This is the result:

My question is this: Is there some activity that goes on when doing a bpy.context.scene.objects.link() that slows things down? I have waited until after the program is run, and all of the objects are created, but the program is still very slow. It takes minutes to interact with it (rotate, zoom, un-hide, etc). You'll notice that the objects are all created with hide=true, so its not trying to display them in the 3D window.

I'm at a loss here, and any help would be appreciated!

Thanks!

EDIT:

The problem is with moving the new cubes from the data-block to the scene. I can run the 'low level' script with any number of cubes up to the 26,000 and it runs very fast and they are all viewable within the data-blocks.

Can anyone help me with a fast way to either duplicate these to the scene or link them to the scene? I've tried both this:

for ob in obs:
    sce.objects.link(ob)

and this:

for j in range(len(obs)):
    sce.objects.link(obs[j])
    bpy.ops.object.duplicate()
    sce.objects.unlink(obs[j])

And they are both very very slow.

Answer 1

Ok - So after messing around with this all day, I came across batFINGER's answer to this persons question (link below- While it wasn't the chosen answer, I was able to modify it to my needs). I don't yet know why it works so fast, but it just does. So thanks batFINGER! The 'collar' here is the same data set as 'cut1' in the question above. This will generate 10,000 objects in just a few seconds.

Creating many primitive objects quickly with Python

Here is the code with my modifications:

import bpy
import pandas as pd
import numpy as np

context = bpy.context
scene = context.scene

collar = collar.as_matrix()

def base_object():
    #Creates base cylinder
    bpy.ops.mesh.primitive_cube_add()
    return

def clear_scene(scene):
    #Clears the current scene
    for obj in scene.objects:
        scene.objects.unlink(obj)
        bpy.data.objects.remove(obj)
    return    

def object_creation(scene, clouds, collar):
    bpy.ops.object.text_add(view_align=False, location=(0,0,0)) 
    txt = bpy.context.object
    base_object()
    obj = scene.objects.active
    obj.parent = txt
    obj.matrix_parent_inverse = txt.matrix_world.inverted()
    mesh = obj.data
    purple = bpy.data.materials.new("Purple")
    purple.diffuse_color = (1.0, 0, 1.0)

    for attempts in range(clouds):
        x = collar[attempts,1]
        y = collar[attempts,2]
        z = collar[attempts,3]
        obj.location = (x, y, z)
        obj.scale = (collar[attempts,4]/2, collar[attempts,5]/2, collar[attempts,6]/2)                  
        obj.show_transparent = True
        obj.active_material = purple
        obj = obj.copy()
        obj.data = mesh.copy() # a new mesh for each object.
        scene.objects.link(obj) # link it to the scene

    return

clear_scene(scene)
object_creation(scene, 10000, collar)