Render large 3D graphs (Millions of vertices & edges)

Question

I need to somehow render several large 3D graphs where each vertex has a position in 3D as well as edges between some of these vertices. I have an XML file with the graph structure (a list of vertices with corresponding id and position as well as a list of edges given as two vertex ids each) and wrote a script to import vertices as spheres and edges as cylinders. This is working fine for up to 100k vertices & edges but importing millions takes way too long (several days).

I am already using low-level operations, creating one sphere (with minimal segment and ring count) and copying that sphere for each vertex in the list. Only afterwards I update the scene:

# Example data:
node_dic = {0: (0,0,0), 1: (0,1,1), 2: (0,1,0), 3: (1,0,0), 4: (1,1,1)}

# Generate base sphere:
bpy.ops.object.select_all(action='DESELECT')
bpy.ops.mesh.primitive_uv_sphere_add(location=(0,0,0), segments=3, ring_count=3)
ob = bpy.context.object
ob.scale = [0.003, 0.003, 0.003]

# Copy object for each item in the dictionary:
for node_id, node_pos in node_dic.items():
    copy = ob.copy()
    copy.data = ob.data.copy()
    loc = np.array(node_pos)
    copy.name = 'node-' + str(node_id)
    generated_nodes[node_id] = copy

# Create vertex group:
group_name = "vertices"    
groups = bpy.data.groups
vertex_group = groups.get(group_name, groups.new(group_name)) 

# Link vertices to group
for ob in list(generated_nodes.values()):
   if ob.name not in vertex_group.objects:
        vertex_group.objects.link(ob)


# Update scene with group instance:
scene = bpy.context.scene
group = bpy.data.groups[group_name]
instance = bpy.data.objects.new(group_name, None)
instance.dupli_type = 'GROUP'
instance.dupli_group = group
scene.objects.link(instance)
scene.update()

I am doing the same for the cylinders representing the edges of the graph with the additional complication that they have to be rotated properly. As mentioned before this doesn't scale.

Thus my question, is there a better way to achieve something like this? A completely different approach maybe or some optimization I am not aware of?

Answer 1

Okay, thanks everybody! What I ended up doing:

1. From the vertex & edge lists create a mesh via python api:

   # Create mesh & object
   mesh = bpy.data.meshes.new("mesh")
   obj = bpy.data.objects.new("graph", mesh)
   
   scene = bpy.context.scene
   scene.objects.link(obj)
   scene.objects.active = obj
   obj.select = True
   
   mesh = bpy.context.object.data
   bm = bmesh.new()
   
   # Example data - replace with parser
   node_dic = {0: (1,0,0), 1: (0,1,0), 2: (0,0,1), 3: (1, 1, 0)}
   edge_list =[(0,1), (1,2), (2,3), (3,0)]
   index_map = {}
   
   # Add vertices to object:
   for node_id, node_pos in node_dic.items():
       pos = np.array(node_pos)
       v = bm.verts.new(pos)
       index_map[node_id] = v 
   
   # Add edges to object:
   for edge in edge_list:
       bm.edges.new((index_map[edge[0]], index_map[edge[1]])) 
   
   bm.to_mesh(mesh)
   bm.free()
   

2. This allows for very fast import of rather large graphs (several million vertices and edges). Now for visualizing the vertices as spheres or anything else use dupliverts (https://docs.blender.org/manual/en/dev/editors/3dview/object/properties/duplication/dupliverts.html) on that object.

3. Visualize the edges between nodes by adding a material to the object.