Rendering on GPU cluster

Question

I'm trying to render a scene on a remote server with GPU cluster, this is the used setup:

• Centos 7
• Blender 2.8
• 4 Nvidia 1080 ti

while rendering I can see that all the GPUs take part in rendering but the utilization is around 33% at peak and the memory usage is low as well around 15% at peak. Rendering one scene takes ~5 minutes while on my laptop computer with single GTX 1060 it takes ~2 minutes to render a single scene. What can be the reason for that? How can I make more efficient usage of my GPU power?

This is the code block used to assign the GPUs:

        preferences = bpy.context.preferences
        cycles_preferences = preferences.addons['cycles'].preferences
        cuda_devices, opencl_devices = cycles_preferences.get_devices()

        cycles_preferences.compute_device_type = 'CUDA'

        for device in cuda_devices:
            if device.type != 'CPU':
                print(f'Activating {device.name}')
                device.use = True
        bpy.data.scenes[0].cycles.device = 'GPU'

The resolution of the scene is really low (300X200) hence expect the rendering to by much quicker than 5 minutes.

Answer 1

Blender can't saturate alone a beefy GPU, let alone 4.

Try using only one GPU at a time.

BTW I work with synthetic datasets (basically render as a map reduce process) and we use an approach kind of like you are showing (Python + BPY). To scale to multiple GPUs and nodes we have to run multiple instances of blender, even on the same GPU. Even though Blender supports CUDA and stuff like that it can't saturate the GPU all the times, so there is a significant time window that the GPU is doing absolutely nothing. Blender is not so VRAM heavy depending on your scene, a few gigs at most but instances have no coordination and aren't so smart to keep VRAM allocated as PyTorch is, so you will see spikes in VRAM usage.

A cluster GPU coordination feature would be neat, like only at most two instances can use GPU full gas at a time.

BTW I already did a render architecture using four 80GB A100 using 20 Blender instances in each one, that few days were crazy lol.