Ok, here's my first draft of an answer. I'll revise it as we talk this out more. If this seems to be going in the right direction, I can add further coding samples.
The key to being able to distribute your render jobs between many render nodes is being able to easily tell the render node what scenes to render.
Currently, from what you've described you would have to add some sort of input argument value to tell the script what scene to load and then render out one frame.
I think if you could modify your script to run on a per frame basis, then map the frame number to scene number, that would make it easier to distribute the job to many machines without having to worry about input values for the scene number.
So the file arrangement would be:
- GenScene.blend file blank file ready for scene setup
- Startup python script ("GenSceneFrame.py") would be run once for every frame of the animation. Let's assume we have 100 scene's to render, we'll set the animation range to 1-100
- SceneData.csv file that contains object setup data for each scene
These 3 files would be packaged together in one folder (or with subfolders if you want but you'd have to be vigilant about paths).
You would have initate the render job from the command line so that you can get it run the python script on startup which is where the frame handler will be created.
It would look something like this:
blender -b GenScene.blend -s 1 -e 100 -a -P GenSceneFrame.py
Let's say for convenience sake that you're going to use a commercial render farm service. They would need to be able to launch your render job with the ability to pipe in the startup python script reference.
You would upload the folder as a Zip file, set the render job for frames 1-100. Then the animation job would be distributed between however many render nodes the service uses, or however many you opt to pay for.
Each frame render runs the internal script and sets up the scene for the current frame number and renders out one frame image. At the end of the job you should have 100 rendered images, one for each scene.