Render 100k objects each with independent color and scale through time

Question

I'm trying to visualize some data that is essentially 100k time series, each with an associated position. I'd like to render this with each object a sphere whose color/scale/alpha is driven by it's time series (something like https://youtu.be/JH6WOUFjAys?t=19 but orders of magnitude bigger).

I currently have a script that populates the scene with UV spheres for each of the objects, inserts all the time series points as keyframes, and uses drivers to share those properties. It works but is incredibly slow to set up. Part of that is inserting all the keyframes, but even duplicating the object that many times is challenging.

I've consider trying to use animation nodes, but don't see how I could pipe in custom data. Particles are the typical answer for this many objects, but I don't see how I can control their position exactly (although I have played with using PLY point cloud) and for sure how to map a time series to each particle.

I'd be happy to write a custom particle system if that's the sensible approach, but can't really find any examples where to start for that.

---

I have a first pass template working with animation nodes (see attached diagram) but find this takes 10s per frame to prepare the scene, even with only 500 objects. I suspect this is because each pass it has to reparse the entire text file for each frame. Would there be a way to cache this or use a binary format? Given each frame corresponds to a sequential row into the text file it seems like there should be a more efficient way.

Answer 1

The most fundamental issue with your node tree is the fact that you included the file parsing node tree inside the loop. So for n number of iterations, you are slowing your node tree by n number of times, in this case 500x ! Furthermore, you need to take advantage of Animation Nodes's victorization abilities, that is, some nodes can process a set of data by default, without the need to use loops. And finally, as you suggested, the data can be cached, and caching is something we can do in Animation Nodes.

---

While CSV can be used, you data set seems to be very big and so it is recommended that your file is in a binary format which you can parse using python and pass the data to Animation Nodes using a script node, alternatively, you can use something like json or yaml.

Parsing

Lets say your CSV file is as follows:

1,1,0,1,8,1,5,7,3,0
6,1,4,1,3,9,6,6,0,7
0,7,8,6,1,2,7,9,5,6
8,4,8,2,0,7,2,4,6,1
4,7,4,6,6,1,0,9,0,3
0,3,8,4,8,3,4,5,0,1
9,2,3,0,8,1,5,2,7,5
2,4,9,5,7,3,4,4,6,0
4,3,1,2,0,6,4,7,6,4
5,9,8,1,6,1,1,8,8,5

Where the first three characters are the x,y,z locations and the rest of the line is the scales at frame 0,1,2, .... This sample file can be parsed as following:

The output Locations vector list is of the same length as the number of objects, where each vector represents the location of one of the objects. However, the Scales vector list is different, it is structured such that the first n number of vectors are the scales of the first object at the frames 0,1,2, ..., the second n number of vectors are the scales of the first object at the frames 0,1,2, ..., and so on. Notice that n is equal to the number of frames there is.

The parsed data is constant, so we can safely cached it. Caching can be enabled through the advanced node settings of the loop:

Instanced Object Transforms

There are multiple ways to generate and position the objects, each have its own limitations, the first of which is by using the Instance Object Node which you used in your node tree:

Using the parsed scale data is not as straight forward as the location data. Notice that the scale of the first object at frame zero is at the index 0, the scale of the second object at frame zero is at the index 0+p where p is the number of frames, the scale of the third object at frame zero is at the index 0+2p and so on. So in general, to get the scales at the frame f of the nth object, we use the general formula f+np. Also, notice that the number of frames is equal to the length of the scales list divided by the number of objects. By using all of previously mentioned facts, we can get the list of scales at some frame using the Slice List Node using the step size as the number of frames:

Notice that we disabled End in the slice node to instruct the node to take steps till the list end. And by animating the scene, we see that it does work as expected:

The problem with this implementation is that it will be relatively slow at higher number of objects, in the next section, I will introduce you to another method to instance tens of thousands of objects with no problem.

DupliFaces

Blender has this feature called DupliFaces, it allows us to instance objects along polygons where the scale of the objects is the area of the polygon and their locations is the centers of the polygons. In Animation Nodes, we can create a mesh composed of triangles such that their areas equals to the scale of the objects and their centers to the locations of the objects, and by using this mesh as a base for dupliFaces and by using a sphere as the parent object, we get exactly what we want:

The Unity Triangle Node is only available in version 2.1, so if you don't have it, just create a mesh with vertices locations equal to:

$$ \begin{bmatrix} -3^{-\frac{1}{4}} \\ -3^{-\frac{3}{4}} \\ 0 \end{bmatrix} \begin{bmatrix} 3^{-\frac{1}{4}} \\ -3^{-\frac{3}{4}} \\ 0 \end{bmatrix} \begin{bmatrix} 0 \\ \left(\frac{2}{3}\right)^{\frac{3}{4}} \\ 0 \end{bmatrix} $$

Notice that to activate duplication, you have to change dupli_type to be FACES, parent of the sphere to be the mesh we created and use_dupli_faces_scale to be True. And that's what we did using the attribute nodes.

A blend file that implements simple Duplicate Groups in AN:

Conclusion

• Don't use loops unless necessary.
• Put anything outside of loops if you can.
• Cache subprogrames if needed.
• Use Duplicate Groups whenever you can.

Coloring Objects

I researched coloring the Duplifaces instances and found that there is no way to color them individually, that is, unless we can decode the random output of the Object Info Node and extract the object index out of it or generate a 3D color map and evaluate the instances locations at it, which is also very hard to do in cycles. ^{_{We really need a better render engine, a render engine we deserve !}}

So our options are limited when it comes to coloring:

• Use a material for each object. Disadvantages: 100k material in a single scene ... I will reject this method.
• Another method would be to use a single material and vertex colors. Disadvantages: Objects have to be Deep Copies, that is, meshes have to be copied as well !
• Set object index for each object and use the Object Index Output from the Object Info Node as texture coordinates for a Nx1 image where N is the number of objects, this texture stores the color of each object in a pixel.

Vertex Colors

This method being simple, it gets slow rapidly, it is 80x slower than our original implementation. Notice that Deep Copy has to be enabled.

Object Index

This method is also simple, but much faster, much lighter and much much more memory efficient. This is due to the fact that we don't Deep Copy the objects and we set only a single integer instead of a full vertex color map.

First step is to set the Pass Index of each object to be 0, 1, 2, 3, ... which represents the index of each object, then we create an image with dimensions Nx1 where N is the number of objects. The first pixel is the color of the first object, the second is the color of the second object and so on. To create this image we use the script:

if imageName in bpy.data.images:
    image = bpy.data.images[imageName]
    image.generated_width = width
    image.generated_height = 1
else:
    image = bpy.data.images.new(imageName, width, 1)

image.pixels = [c for color in colors for c in color]

In the material side, we normalize the indices to represents texture coordinates for an image by dividing by the number of objects. The implementation is as simple as:

Notice that if the number of objects is constant, you may removed the Object Attribute Output node because the objects already have their indices set, this will reduce the execution by almost half! Also, note that we can automatically set the value of the divide node through animation nodes using the Cycles Material Output Node:

I think it is obvious which method you should be using.