# Render 100k objects each with independent color and scale through time

I'm trying to visualize some data that is essentially 100k time series, each with an associated position. I'd like to render this with each object a sphere whose color/scale/alpha is driven by it's time series (something like https://youtu.be/JH6WOUFjAys?t=19 but orders of magnitude bigger).

I currently have a script that populates the scene with UV spheres for each of the objects, inserts all the time series points as keyframes, and uses drivers to share those properties. It works but is incredibly slow to set up. Part of that is inserting all the keyframes, but even duplicating the object that many times is challenging.

I've consider trying to use animation nodes, but don't see how I could pipe in custom data. Particles are the typical answer for this many objects, but I don't see how I can control their position exactly (although I have played with using PLY point cloud) and for sure how to map a time series to each particle.

I'd be happy to write a custom particle system if that's the sensible approach, but can't really find any examples where to start for that.

I have a first pass template working with animation nodes (see attached diagram) but find this takes 10s per frame to prepare the scene, even with only 500 objects. I suspect this is because each pass it has to reparse the entire text file for each frame. Would there be a way to cache this or use a binary format? Given each frame corresponds to a sequential row into the text file it seems like there should be a more efficient way.

• I uses animation nodes to achieve similar results. How is the data represented? Are color data discrete data or can be described somehow? – Omar Emara Feb 19 '18 at 20:19
• right now I have the time series as a csv file but can easily reformat it. there is basically one time series per object and then i map that to scale, alpha, emission, and object R,G,B. More specifically i read my CSV file and create an F-curve/keyframes on red, then use drivers to map it to the other fields. – peabody124 Feb 19 '18 at 20:21
• Can you provide a sample CSV file with couple of objects? You can just write it here as a code block. I will have to go now, but I will answer your question tomorrow as soon as you provide a sample. – Omar Emara Feb 19 '18 at 20:26
• I can, but at this point the format of my CSV is quite awkward to try and sparse the keyframes so would not be a good resource. I'm looking more for guidance how to go about this but can manage the low level details. If you can point me to any tips/examples of piping external data into an animation node that would be great. Are animation nodes ultimately just manipulating lower level Blender structures and providing a visual means to wire them up? – peabody124 Feb 19 '18 at 21:40
• Are you after an animation nodes answer only? If not, can you share your "script that populates the scene with UV spheres" How are you duplicating them? Using foreach_set to build the fcurves from the data, and perhaps converting to sampled points,... among others could make it faster. – batFINGER Feb 20 '18 at 15:20

The most fundamental issue with your node tree is the fact that you included the file parsing node tree inside the loop. So for n number of iterations, you are slowing your node tree by n number of times, in this case 500x ! Furthermore, you need to take advantage of Animation Nodes's victorization abilities, that is, some nodes can process a set of data by default, without the need to use loops. And finally, as you suggested, the data can be cached, and caching is something we can do in Animation Nodes.

While CSV can be used, you data set seems to be very big and so it is recommended that your file is in a binary format which you can parse using python and pass the data to Animation Nodes using a script node, alternatively, you can use something like json or yaml.

# Parsing

Lets say your CSV file is as follows:

1,1,0,1,8,1,5,7,3,0
6,1,4,1,3,9,6,6,0,7
0,7,8,6,1,2,7,9,5,6
8,4,8,2,0,7,2,4,6,1
4,7,4,6,6,1,0,9,0,3
0,3,8,4,8,3,4,5,0,1
9,2,3,0,8,1,5,2,7,5
2,4,9,5,7,3,4,4,6,0
4,3,1,2,0,6,4,7,6,4
5,9,8,1,6,1,1,8,8,5


Where the first three characters are the x,y,z locations and the rest of the line is the scales at frame 0,1,2, .... This sample file can be parsed as following:

The output Locations vector list is of the same length as the number of objects, where each vector represents the location of one of the objects. However, the Scales vector list is different, it is structured such that the first n number of vectors are the scales of the first object at the frames 0,1,2, ..., the second n number of vectors are the scales of the first object at the frames 0,1,2, ..., and so on. Notice that n is equal to the number of frames there is.

The parsed data is constant, so we can safely cached it. Caching can be enabled through the advanced node settings of the loop:

# Instanced Object Transforms

There are multiple ways to generate and position the objects, each have its own limitations, the first of which is by using the Instance Object Node which you used in your node tree:

Using the parsed scale data is not as straight forward as the location data. Notice that the scale of the first object at frame zero is at the index 0, the scale of the second object at frame zero is at the index 0+p where p is the number of frames, the scale of the third object at frame zero is at the index 0+2p and so on. So in general, to get the scales at the frame f of the nth object, we use the general formula f+np. Also, notice that the number of frames is equal to the length of the scales list divided by the number of objects. By using all of previously mentioned facts, we can get the list of scales at some frame using the Slice List Node using the step size as the number of frames:

Notice that we disabled End in the slice node to instruct the node to take steps till the list end. And by animating the scene, we see that it does work as expected:

The problem with this implementation is that it will be relatively slow at higher number of objects, in the next section, I will introduce you to another method to instance tens of thousands of objects with no problem.

# DupliFaces

Blender has this feature called DupliFaces, it allows us to instance objects along polygons where the scale of the objects is the area of the polygon and their locations is the centers of the polygons. In Animation Nodes, we can create a mesh composed of triangles such that their areas equals to the scale of the objects and their centers to the locations of the objects, and by using this mesh as a base for dupliFaces and by using a sphere as the parent object, we get exactly what we want:

The Unity Triangle Node is only available in version 2.1, so if you don't have it, just create a mesh with vertices locations equal to:

$$\begin{bmatrix} -3^{-\frac{1}{4}} \\ -3^{-\frac{3}{4}} \\ 0 \end{bmatrix} \begin{bmatrix} 3^{-\frac{1}{4}} \\ -3^{-\frac{3}{4}} \\ 0 \end{bmatrix} \begin{bmatrix} 0 \\ \left(\frac{2}{3}\right)^{\frac{3}{4}} \\ 0 \end{bmatrix}$$

Notice that to activate duplication, you have to change dupli_type to be FACES, parent of the sphere to be the mesh we created and use_dupli_faces_scale to be True. And that's what we did using the attribute nodes.

A blend file that implements simple Duplicate Groups in AN:

# Conclusion

• Don't use loops unless necessary.
• Put anything outside of loops if you can.
• Cache subprogrames if needed.
• Use Duplicate Groups whenever you can.

# Coloring Objects

I researched coloring the Duplifaces instances and found that there is no way to color them individually, that is, unless we can decode the random output of the Object Info Node and extract the object index out of it or generate a 3D color map and evaluate the instances locations at it, which is also very hard to do in cycles. We really need a better render engine, a render engine we deserve !

So our options are limited when it comes to coloring:

• Use a material for each object. Disadvantages: 100k material in a single scene ... I will reject this method.
• Another method would be to use a single material and vertex colors. Disadvantages: Objects have to be Deep Copies, that is, meshes have to be copied as well !
• Set object index for each object and use the Object Index Output from the Object Info Node as texture coordinates for a Nx1 image where N is the number of objects, this texture stores the color of each object in a pixel.

## Vertex Colors

This method being simple, it gets slow rapidly, it is 80x slower than our original implementation. Notice that Deep Copy has to be enabled.

## Object Index

This method is also simple, but much faster, much lighter and much much more memory efficient. This is due to the fact that we don't Deep Copy the objects and we set only a single integer instead of a full vertex color map.

First step is to set the Pass Index of each object to be 0, 1, 2, 3, ... which represents the index of each object, then we create an image with dimensions Nx1 where N is the number of objects. The first pixel is the color of the first object, the second is the color of the second object and so on. To create this image we use the script:

if imageName in bpy.data.images:
image = bpy.data.images[imageName]
image.generated_width = width
image.generated_height = 1
else:
image = bpy.data.images.new(imageName, width, 1)

image.pixels = [c for color in colors for c in color]


In the material side, we normalize the indices to represents texture coordinates for an image by dividing by the number of objects. The implementation is as simple as:

Notice that if the number of objects is constant, you may removed the Object Attribute Output node because the objects already have their indices set, this will reduce the execution by almost half! Also, note that we can automatically set the value of the divide node through animation nodes using the Cycles Material Output Node:

I think it is obvious which method you should be using.

• Awesome, thank you. That runs much more reasonably. It can parse 3000 cells in 200ms. I couldn't quite get the last part (duplicate groups) working perfectly - although I can directly take the mesh from a Icosphere and replicate that and it seems to work. However, it actually runs a bit slower than the first version (scaling just the triangle takes ~150ms so only a small gain with parenting not working). however, is there a way to vectorize setting the colors of the objects too? – peabody124 Feb 20 '18 at 17:53
• @peabody124 Surely something is not right, I can replicate a million triangle in just 40ms. What exactly isn't working for you in the in the DupliGroup method? – Omar Emara Feb 20 '18 at 18:23
• @omarsquircleart (btw I really liked your blog posts). Here is my diagram imgur.com/a/Wu7l9 which shows it taking 130ms for 3k objects. in addition you can see from the top view that the triangles have varying sizes but the Icosphere parented to them doesn't take on the same size. – peabody124 Feb 20 '18 at 18:45
• @peabody124 I see where things went wrong with the scaling. It turns out we have to enable use_dupli_faces_scale in another attribute node. I added a blend file that implements a simplified implementation of the duplicate group. It takes only 1ms on my machine, can you test it and tell me how much it takes for you? The unity triangle node is not in your AN version, so use the mesh you uses before. Also, below the Execute Node Tree button, there is a label called Details, can you press it and send me another screen shot? – Omar Emara Feb 20 '18 at 19:07
• Oh, I need to get AN 2.1 -- is there a place where I can download it already compiled or will i have to do that myself? – peabody124 Feb 20 '18 at 19:19