I've got a Ubuntu 14.04 headless server running right now for 3d rendering on Blender's Cycles rendering engine, across 4 GPUs.
(4) ASUS GTX 970s, 4GB 1300W PSU Core i7 16GB RAM 250GB SSD
Animations are rendering beautifully across the 4 GPUs in Cycles, but my question is concerning parallelizing processes to those cards:
Is it possible for me to have 4 separate blender processes running in parallel targeting one specific GPU each?
Running all 4 cards as "one" card (
CUDA_MULTI_0 in a .py setup) is not actually a full 4x speed increase over running just one card, though it is still quite fast, and amazing i might add having come from mostly CPU rendering. Anyway, it turns out to be more like 3x speed, and i want to maximize render times further by rendering one animation frame per card in parallel, giving me a true 4x speed.
Theoretically, it should just be a matter of running 4 separate blender processes on the command line, while having python scripts setup to target specific cards like this:
$ blender -b file.blend -P cuda1.py -s 1 -e 10 -a & $ blender -b file.blend -P cuda2.py -s 11 -e 20 -a & $ blender -b file.blend -P cuda3.py -s 21 -e 30 -a & $ blender -b file.blend -P cuda4.py -s 22 -e 40 -a &
and the cuda[x].py scripts look like this, where each one is targeting a specific CUDA device:
import bpy, _cycles bpy.context.scene.cycles.device = 'GPU' bpy.context.user_preferences.system.compute_device_type = 'CUDA' # this is different in each cuda[x].py file, CUDA_0, CUDA_1, CUDA_2, CUDA_3 bpy.context.user_preferences.system.compute_device = 'CUDA_0'
So, this all technically works right now, but here's the big issue i'm trying to figure out. When i do this, only the first blender process gets GPU access. Processes 2-4 default back the CPU, ignoring the GPU instructions. All four cards work just fine, and as I mentioned, one blender process using the
CUDA_MULTI_0 device in the python script is leveraging all four cards as expected.
Is there some known limitation to blender, or CUDA/NVIDIA for that matter, that only lets a single processes access any or all the GPUs at one time, that maybe i'm just not aware of?
Any advice on this is greatly appreciated.