What is the fastest way of creating a large mesh/object built from smaller mesh "building blocks"

Question

I realize similar questions have been asked before and I followed the advice given there but I hope to gain more insight on the matter here.

I've added a lot of details, I hope it's not too much.

I have information about voxel-based terrain, which I want to put together in a 3D model, basically using different voxel building blocks. The currently fastest version of my script will create 980000 vertices (after removing doubles) in 5 minutes.

The relevant parts of my script look something like this (simplified to focus on relevant parts):

generate_relevant_map_data_from_file()

bm = bmesh.new()
stepsize = 500
col_num = 0

#this part builds a dictionary for the different voxel types with the corresponding
#bmesh and translocation vector
obj_meshes = ["WALL", "FLOOR", "TREE", ...]            
mesh_dict = {s: type('bm_ext', (object,), dict(v=Vector((0, 0, 0)), bm=bmesh.new())) for s in obj_meshes}
for k in mesh_dict:
    mesh_dict[k].bm.from_object(bpy.data.objects[k], bpy.context.scene)

#this is where all the stuff is generated
for c in columns:
    col_num += 1
    for t in c:
        identificator = t.generate_id()
        loc = Vector((t.x * 2, t.y * - 2, t.z * 3))

        #this is where I tried to optimize things
        #the different attempts are shown and explained below
        """
        INSERT CODE HERE
        """

    if numBlocks >= stepsize:
        bmesh.ops.remove_doubles(bm, verts=bm.verts, dist=0.0001)
        me = bpy.data.meshes.new("landscape")
        bm.to_mesh(me)
        ob = bpy.data.objects.new("Land", me)
        bpy.context.scene.objects.link(ob)
        bm = bmesh.new()
        numBlocks = 0
        bpy.ops.wm.save_mainfile(filepath=filename)
        bpy.ops.wm.open_mainfile(filepath=filename)

#do the same as above to finalize object creation

The fastest version for INSERT CODE HERE I have found so far looks like this:

add_bm = mesh_dict[identificator].bm
bmesh.ops.translate(add_bm, vec=loc-mesh_dict[identificator].v, verts=add_bm.verts)
mesh_dict[identificator].v = loc
me = bpy.data.meshes.new("temp")
add_bm.to_mesh(me)
bm.from_mesh(me)

First of all, I noticed that the speed is very dependent on how much memory Blender is using. Even when I just interrupt the script, delete anything it created and restart it, the speed will be similar to when I interrupted it. This is why I reload the .blend file every 500 columns. Is there a better way to do this?

Second: Is there a function to directly copy bmesh data from one to another? Right now I first export to Mesh, and then load from there, which seems really clunky.

If nothing else, it would at least be nice to these 2 things explained/resolved.

Some of my other attempts are described below.

At first, I didn't have the dictionary with the bmeshes and created them new every time:

add_bm = bmesh.new()
add_bm.from_object(bpy.data.objects[identificator], bpy.context.scene)
bmesh.ops.translate(add_bm, vec=loc, verts=add_bm.verts)
me = bpy.data.meshes.new("temp")
add_bm.to_mesh(me)
add_bm.free()
bm.from_mesh(me)

It is slightly slower than what I have now, but not by much (286s vs 292s, so maybe it was just chance).

I think this was one of my earliest attempts:

start_vert = len(bm.verts)
bm.from_object(bpy.data.objects[identificator], bpy.context.scene)
bmesh.ops.translate(bm, vec=loc, verts=bm.verts[start_vert:len(bm.verts)])

It already took 58 sec for the first batch of 500 columns, when there is a total of 9000.

I thought, maybe always creating the new meshes was taking a lot of time and I couldn't find a function to copy data directly from one bmesh to another so I tried this:

add_bm = bmesh.new()
add_bm.from_object(bpy.data.objects[identificator], bpy.context.scene)
bmesh.ops.translate(add_bm, vec=loc, verts=add_bm.verts)
vertindex_offset = len(bm.verts)
for v in add_bm.verts:
    bm.verts.new(v.co)
for e in add_bm.edges:
    bm.edges.new([bm.verts[v.index + vertindex_offset] for v in e.verts])
for f in add_bm.faces:
    bm.faces.new([bm.verts[v.index + vertindex_offset] for v in f.verts])
add_bm.free()

I thought this would be a more low-level approach, but I was obviously very mistaken, because this one also already took 60 seconds for the first batch of 500 columns.

UPDATE: I also tried creating a bunch of linked object copies like this:

copy = bpy.data.objects[identificator].copy()
copy.location = loc
bpy.context.scene.objects.link(copy)

But it was also a lot slower. Additionally, opening the file takes several seconds.

Answer 1

You might try to use Dupli Verts:

• Create your mesh object

  mesh = bpy.data.meshes.new("WorldLocations")
  mesh.vertices.add(number_of_vertices)
  #flat_vertex_coordinates x0, y0, z0, x1, y1, z1, ...
  mesh.vertices.foreach_set("co", flat_vertex_coordinates)
  
  duplicator = bpy.data.objects.new("Duplicator", mesh)
  bpy.context.scene.objects.link(duplicator)
  

• Parent your object to be duplicated:

  original = bpy.data.objects[original]
  original.parent = duplicator
  

• Set the duplication type

  duplicator.dupli_type = 'VERTS'
  

Instead of bmesh you might also try Mesh.from_pydata. The source code of this function is located at scripts/modules/bpy_types.py

Answer 2

pink vertex' idea using from_pydata works, I just wanted to add a few more details.

my code:

add_bm = mesh_dict[identificator].bm
bmesh.ops.translate(add_bm, vec=loc-mesh_dict[identificator].v, verts=add_bm.verts)
mesh_dict[identificator].v = loc
#until here identical, then I copy the raw data into python lists instead of directly
#into the bmesh
vertindex_offset = len(all_v)
all_v.extend(v.co[:] for v in add_bm.verts)
all_f.extend([[v.index + vertindex_offset for v in f.verts] for f in add_bm.faces])

Then outside the main loop I do the mesh.from_pydata(all_v, [], all_f) and create the object etc.

This is more than twice as fast than my previously fastest version, gets around needing to reload the repeatedly, has a larger memory overhead though (mainly because I remove all doubles at once at the end, which could be optimised).

It is essentially the "lowlevel" approach I tried, but apparently the bmesh.verts.new() is pretty slow (I suspect the entire array is copied every time) and there doesn't seem to be a method to add an entire list of verts to a bmesh (yet).

edit: The DupliVerts method is by far the fastest method I've tried so far (probably 10 times as fast as from_pydata). I'm not sure, however, if I will be able to use it for my particular problem.