Enabling GPU rendering on headless linux machine

Question

I'm trying to enable GPU rendering on a headless linux machine. I'm using the following code:

            print("----------------------------------------------")
            print('setting up gpu ......')

            bpy.context.scene.cycles.device = "GPU"
            for scene in bpy.data.scenes:
                print(scene.name)
                scene.cycles.device = 'GPU'

            bpy.context.preferences.addons["cycles"].preferences.compute_device_type = "CUDA"

            bpy.context.preferences.addons["cycles"].preferences.get_devices()
            print(bpy.context.preferences.addons["cycles"].preferences.compute_device_type)

            bpy.context.preferences.addons["cycles"].preferences.get_devices()
            for d in bpy.context.preferences.addons["cycles"].preferences.devices:
                d.use = True
                if d.type == 'CPU':
                    d.use = False
                print("Device '{}' type {} : {}".format(d.name, d.type, d.use))
            print('setting up gpu done')
            print("----------------------------------------------")

and I got the result showing as follows:

----------------------------------------------
setting up gpu ......
Scene
CUDA
Device 'NVIDIA TITAN Xp' type CUDA : True
Device 'Intel Xeon CPU E5-2620 v4 @ 2.10GHz' type CPU : False
Device 'NVIDIA TITAN Xp' type OPTIX : True
setting up gpu done
----------------------------------------------

However, I found the rendering speed still very slow (similar to CPU rendering), and nvidia-smi showed that no processes were running.

My blender version is 3.3. Is anything wrong with my setup?

Answer 1

This could be because you have CUDA libraries installed, but not the nvidia drivers. When you run blender what is the output of nvidia-smi? Do you see the blender/python process listed? If the nvidia-smi command isn't found that would indicate you are likely missing the nvidia drivers and need to install them to take advantage of your GPU. As far as I can tell, if the CUDA libraries are present and the GPU is identifiable Blender will silently fall back to using the CPU even though the GPU shows up in get_devices() and you set your compute_device_type to 'CUDA'.

Specifically, I had this same problem running blender on a google cloud kubernetes cluster with a GPU and the issue was that I hadn't added

resources:
  limits:
    nvidia/gpu: 1

to my container yaml config. Apparently without this google cloud doesn't mount the exposed volumes with the nvidia drivers. So even though I was building a container FROM nvidia/cuda:11.4.2-base-ubuntu20.04 which makes sure all CUDA libraries are present and the system could identify the GPU with lshw, I didn't have access to the nvidia drivers so Blender just happily used the CPU instead. (Note the volume with the binaries only exists if you've installed the nvidia drivers in your cluster https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers)