More efficient usage of the computing resources by running several Blender instances

Question

I'm using Eevee to render a mesh sequence imported with Stop Motion OBJ in "Streaming" mode. I only have 1 or 2 such mesh sequence in the scene. I play the sequence(s) while the camera is moved around according to camera poses read from some pre-recorded traces (CSV files). The camera poses in the trace files are generated from another source. Then I render and save the result for each frame as a PNG image. I also apply some image processing and do calculations on the rendered images using the Python interface.

Now I'm doing this for several such traces and for different mesh sequences, so I have to perform hundreds of runs/experiments and render each frame. My problem is: if I do this one-by-one for each trace, only a small amount of CPU and memory resources are used, and the amount of time necessary for all experiments is unacceptable. I'm already rendering at the minimum possible resolution and cannot seem to get the rendering faster by further optimization of the renderer parameters.

I found sort of a workaround by starting multiple Blender instances in parallel in background mode (on the same machine) and distributing the traces to different instances i.e. data parallelism. To do this, I simply start a tmux session and run multiple instances of Blender from the command line, in different tmux windows. However, still the CPU/memory resources are not even halfway exhausted on my 64-core, 32GB RAM machine. I think that the best way to speed up the whole thing is to start as many instances of Blender as the number of different camera traces, until I hit the boundaries of CPU or memory resources.

Does my approach make sense? Is there any more elegant way of scaling up the rendering for such simple scenes that consume very few resources?

Answer 1

I ended up using subprocess.Popen to run the whole thing in a more automatized way. Here is the relevant part of my Python script, in case someone needs it in the future:

data_dir = /path/to/csv_files
f_list = glob.glob(str(data_dir / "*.csv"))
cmds_list = [[blender, '--background', 'my_scene.blend',
                  '--python', 'scripts/process_data.py', '--',
                  '-t', fname] for fname in f_list]
   
    
# stdout should be redirected to /dev/null, otherwise filling up the OS pipe buffers and blocking
procs_list = [sp.Popen(cmd, stdout=sp.DEVNULL, stderr=sp.STDOUT) for cmd in cmds_list]
for proc in procs_list:
    proc.wait()

After a little experimenting, I figured out the necessary amount of processes to almost fully utilize my CPU and memory resources, so I tweak the batch size in data_dir accordingly.