Rendering multiple scenes in parallel

Question

I have 100s of scenes with simple primitive objects at random locations (not in a sequence) in each of these scenes. For example if I have 2 objects in a scene, the only thing that will vary across all 100s of scenes is the location, orientation of these 2 objects. I have file generated with all these locations and orientations to load into blender from. Currently in my project I am sequentially loading each location, orientation detail of a scene and rendering it one after the other.

Is there a way to generate these 100 scenes in parallel? I have been looking at GPU rendering/cluster based rendering but failed to find any concrete that I can use.

Any pointer will be appreciated. Thanks in advance!

Answer 1

Ok, here's my first draft of an answer. I'll revise it as we talk this out more. If this seems to be going in the right direction, I can add further coding samples.

The key to being able to distribute your render jobs between many render nodes is being able to easily tell the render node what scenes to render.

Currently, from what you've described you would have to add some sort of input argument value to tell the script what scene to load and then render out one frame.

I think if you could modify your script to run on a per frame basis, then map the frame number to scene number, that would make it easier to distribute the job to many machines without having to worry about input values for the scene number.

So the file arrangement would be:

• GenScene.blend file blank file ready for scene setup
• Startup python script ("GenSceneFrame.py") would be run once for every frame of the animation. Let's assume we have 100 scene's to render, we'll set the animation range to 1-100
• SceneData.csv file that contains object setup data for each scene

These 3 files would be packaged together in one folder (or with subfolders if you want but you'd have to be vigilant about paths).

You would have initate the render job from the command line so that you can get it run the python script on startup which is where the frame handler will be created.

It would look something like this:

blender -b GenScene.blend -s 1 -e 100 -a -P GenSceneFrame.py

Let's say for convenience sake that you're going to use a commercial render farm service. They would need to be able to launch your render job with the ability to pipe in the startup python script reference.

You would upload the folder as a Zip file, set the render job for frames 1-100. Then the animation job would be distributed between however many render nodes the service uses, or however many you opt to pay for.

Each frame render runs the internal script and sets up the scene for the current frame number and renders out one frame image. At the end of the job you should have 100 rendered images, one for each scene.

Answer 2

As you have a script creating all the scenes, I can think of three approaches, the render farm or way you distribute the render will influence which suits you.

One is to save a file after each scene is created. At the end of your scene creation loop, instead of rendering you save the scene out to it's own file - bpy.ops.wm.save_as_mainfile(filepath='scene'+str(index)). These individual files can be submitted as separate tasks to the render farm that you are using.

Another is to submit each render task after it is created. This will depend on the render farm you are using, if there is an addon that allows you to submit the task from blender then you can add that operator to your script, instead of bpy.ops.render.render()

The third approach would be to alter your script to only contain the tasks you have inside your loop. Use sys.argv to get a variable passed in when starting blender from the cli and use it as the index value instead of an index range as you loop through. Each machine that starts rendering will be given a different value or range of values to render.

blender -b --python myscript.py -- 5

myscript.py could contain -

import bpy
from sys import argv

SceneIdx = argv[argv.index('--') + 1]

mydata = ReadData('file.dat')
bpy.data.objects['Cube'].location = mydata[SceneIdx]
bpy.ops.render.render()