How can I use the blender python API to make a scatter plot animation of millions of objects?

Question

I have produced some data, that I would like to visualize with some eye candy. The data is contained in a 3D numpy array a. a[t,j,k] would be the k-th component of the position of the j-th object at time t. Though I am flexible changing that if need be.

I thought of two ways of implementing this. One is a script that creates instances of a mesh and writes keyframes for them but that implementation uses python for loops and since I want an animation for about one minute this will take too long. Using multiprocessing or just multi threading wasn't working as soon as I touched anything from bpy. Perhaps using a particle system will be more suitable for this task. Here I also an instance object. I found some code from the post How can I create many objects quickly? but the API seemed to change over time, so it doesn't quite work. I rewrote what I could find only random initialization is possible.

def create_many_copies(target_obj, count):
    # Create a particle system container
    bpy.ops.mesh.primitive_cube_add()
    cube = bpy.context.object

    # Create a particle system within the container
    bpy.ops.object.particle_system_add()
    particle_sys = cube.particle_systems[0].settings

    # Set parameters so all particles show up at once
    particle_sys.frame_start = -1
    particle_sys.frame_end = 1

    particle_sys.count = count

    particle_sys.emit_from = 'VOLUME'

    # No random rotations
    particle_sys.use_rotations = True
    particle_sys.rotation_mode = 'NONE'

    # No physics
    particle_sys.physics_type = 'NO'

    # Use instance object for particle
    particle_sys.render_type = 'OBJECT'
    particle_sys.instance_object = target_obj
    particle_sys.particle_size = 1.0
    particle_sys.use_rotation_instance = True
    particle_sys.use_scale_instance = True

    # Make sure we're at the correct frame
    bpy.context.scene.frame_set(1) 

    # Looping over new position vectors
    # This results in an error telling me I am supposed to give it an
    # array of length 0, but gave array with length 30. Here 30 = len(locs).

    locs = np.arange(count * 3)
    cube.particle_systems[0].particles.foreach_set('location', locs)


bpy.ops.mesh.primitive_cone_add()
target = bpy.context.object 
create_many_copies(target, 10)

I don't really mind random initialization nor do I need the locations of the particles, as long as I can animate the locations using keyframes anyway. I would really appreciate a solution to keyframing the particles using the .foreach_set() method. If all this is too cumbersome and not 'the right way', then please tell me.

If it is possible to just forgo running a script in blender at all, I am all for it. Can I perhaps use python to 'bake' data and use it as external cache for a particle system inside blender? Any help is appreciated!

Answer 1

I've had similar problems and ended up creating a python-only addon called blender-plots which provides animated scatterplots. I've managed to avoid looping so far using these approaches:

• Creating many copies of the same object: create a mesh with vertices at each point you want an object and use an Instance on Points node to set the appearance
• Creating mesh: use mesh.from_pydata(vertices, [], [])
• Updating vertices on existing mesh: use mesh.vertices.foreach_set("co", vertices.reshape(-1)) (vertices = Nx3 numpy array)
• Animating vertices: See below

To create an animation, first load the vertices for all timestamps as a single mesh, then add a frame_index attribute to each vertex (frame_index_per_vertex = length-N numpy array):

mesh.attributes.new(name="frame_index", type="FLOAT", domain="POINT")
mesh.attributes[attribute_name].data.foreach_set("value", frame_index_per_vertex)

Then add a geometry node modifier set up to only show the vertices where frame_index matches the current frame: The first Value field in the Compare node is keyframed to interpolate between 1 and n_frames. If you use the addon all this is set up automatically :)

It would also be possible to keyframe the individual vertex positions directly to get proper interpolation (with the option above objects will make sharp jumps between frames), see this question. However afaik it would require looping over every vertex with the current API.