Linux cmdline rendering on multiple GPUs

Question

I'm running a number of benchmarks on Linux (Ubuntu 18.04) cloud VMs that have multiple GPUs attached, however I can only ever render on a single GPU. I'm not using the standard Blender benchmark executable as I kept running into checksum errors, rather I'm installing Blender locally and following the instructions to run offline benchmarks here.

I've tried adding this script to my command line to force-enable all my GPUs, but I'm still only getting a single GPU rendering. My command line is:

  /usr/bin/blender \
    --background \
    --factory-startup \
    -noaudio \
    --enable-autoexec \
    --engine \
    CYCLES \
    /path/to/scenes/main.blend \
    --python \
    /path/to/scripts/use_gpus.py \
    --python \
    /path/to/scripts/main.py \
    -- \
    --device-type=CUDA

Am I missing something? Is the GPU device ID baked into the benchmark scenefiles, not to be overridden?

Answer 1

The benchmark script sets the enabled devices itself, hence the script to enable all GPUs has no effect. Since the purpose of the benchmark is to evaluate the performance of a specific CPU or GPU, it only enables one (compute) device at a time. This can be seen by taking a look at the benchmark scripts.

In main.py it first parses the supplied arguments and starts the benchmark. You can specify which device shall be used through the optional -d, --device argument. The _benchmark() function gets the requested device as an argument and calls the render() function from the render.py script. It enables the requested device either through _enable_device() which in turn calls _enable_compute_device() or _enable_cpu_device(), depending whether the benchmark should be performed for a GPU or CPU. In your case it would be using the former function. _enable_compute_device() first disables all enabled graphics cards. If you supplied a specific GPU, it will try to enabled it. If no GPU was given as an argument (requested_device.name is None), it first tries to find one that isn't used as display device and if no such GPU exists, it picks the first one available. In both cases it only enables one GPU and it's the first one that matches the criteria.

Therefore you need to call the benchmark script multiple times and supply the GPU name through the --device argument for each run. The available device names can be printed to the terminal by using the -l or --list-devices argument.

Alternatively you would have to modify the script. However, this would also break the JSON reporting function since it expects exactly one device to be enabled. The report format could be modified as well, but it would not be possible to upload the modified reports to the Open Data platform.

Answer 2

To make this work, I modified the script by @robert-gützkow. It's a bit hacky, but it allows the render to return proper JSON structured output.

As you iterate over each device, cast each returned GPU into a ComputeDevice thusly:

ComputeDevice(
  name=device.name,
  type=DeviceType('CUDA'),
  is_display=False,
  cycles_device=device
)

Then, in render.py in the scripts dir, I replaced:

enabled_devices = _enable_device(requested_device)

with

enabled_devices = enable_gpus.enable_gpus('CUDA')