Blender sets Compute Device (CUDA) but doesn't use it for actual render (on EC2 GPU Instance)

Question

I'm trying to set up some automated render servers on EC2 using their GPU instance type (g2.2xlarge). I'm running blender in the background with a python script that renders the image I want and saves it to the right place in the file system. I'm using Cycles, and blender 2.69 on ubuntu server 12.04.3 (for HVM instances).

Background: Running the script works fine and the image is successfully (and correctly) rendered and stored on the system.

I'd like to use the GPU though for added speed. I've set the GPU in python like so:

bpy.context.user_preferences.system.compute_device_type = 'CUDA'
bpy.context.user_preferences.system.compute_device = 'CUDA_0'

I know this succeeds because if I put dummy data in there (say "GPU" in the first one), it fails at that point and says that it can only be in ("CPU", "CUDA"). Similar thing for the second option.

I also know cuda is successfully installed and running, as I have no problems with nvidia-smi and the deviceQuery cuda sample.

The problem: When I do run it, and observe the cpu with htop and the gpu with nvidia-smi -l, I see all the CPU cores at 100% but nothing in nvidia-smi. I also see no notable difference in time from running it without setting any GPU settings in python. So I'm fairly convinced that it's accepting the settings and then not running the render on the GPU.

*Note: I'm newish to blender so perhaps there's another set of settings I need to turn on to get the actual render to happen on the GPU? I'm using

bpy.ops.render.render(write_still=True)

to render the image, it that helps anyone.

Any help would be much appreciated :) Also if I haven't described something enough, I'm happy to go into more detail on the setup.

Update:

I ran a separate blend file: http://dl.dropbox.com/u/1742071/1m/BMW1M-MikePan.blend that I got from http://blenderartists.org/forum/showthread.php?239480-2-61-Cycles-render-benchmark

I'm running it with:

import bpy

bpy.context.user_preferences.system.compute_device_type = 'CUDA'
bpy.context.user_preferences.system.compute_device = 'CUDA_0'

bpy.ops.render.render(write_still=True)
img_path = "/mnt/render-output/bmw.jpg"
rendered_image = bpy.data.images["Render Result"]
rendered_image.save_render(filepath=img_path)

This does work and successfully renders an image using the GPU (I can see it in nvidia-smi).

So my question is now: What could be wrong in my blend file / script that would make it switch back to cpu rendering? Are there limits on textures/vertices/something that would make it switch back to the CPU with no warning / error messages? Is there a way to run blender such that it'll fail if it can't run it on the GPU?

Update 2017-04-16:

Cycles CUDA settings (Blender 2.78c) appear to have changed in the API from:

bpy.context.user_preferences.system.compute_device_type = 'CUDA'
bpy.context.user_preferences.system.compute_device = 'CUDA_0'

to:

bpy.context.user_preferences.addons['cycles'].preferences.compute_device_type = 'CUDA'
bpy.context.user_preferences.addons['cycles'].preferences.devices[0].use = True

Answer 1

I found one more setting to check that sets Blender to use CPU. I rendered using this script(gpurender.py):

import bpy
bpy.context.scene.cycles.device = 'GPU'
bpy.ops.render.render(True)

Command-line was:

blender -b file.blend -E CYCLES -t 0 -o `pwd`/outpre -P gpurender.py

If you check in python, bpy.context.scene.cycles.device will probably be CPU. I've tested and this uses the GPU on my EC2 instance.

Answer 2

I had a similar problem. I installed Debian on an old PC to be a blender render server, but even after apt-get install nvidia-driver nvidia-cuda-toolkit (which was its own bucket of spiders) blender wasn't using my GPU.

I decided I wanted the GPU render to be the default, so I created this script and ran it from the command line:

import bpy

prefs = bpy.context.user_preferences.addons['cycles'].preferences
prefs.compute_device_type = 'CUDA'
prefs.compute_device = 'CUDA_0'

bpy.ops.wm.save_userpref()

Then blender -b -P /var/tmp/gpurender.py to run it and save the preferences for future compute jobs. If you have access to X11 you can do the same thing interactively, but I didn't bother to install any windowing system on that box.

The API change from Headless render: can't find compute_device in 2.78b is already noted at the end of the original question