If I reset Blender to factory settings, turn on CUDA + Cycles, delete everything in the scene but the camera, and hit F12 to render, I get an Out of Memory error:
CUDA error: Out of memory in cuLaunchKernel(cuPathTrace, xblocks, yblocks, 1, xthreads, ythreads, 1, 0, 0, args, 0)
I've already made sure of the following things:
My GPU [512MB NVIDIA GeForce GT 640M] supports CUDA and has a 3.0 compute capability (more than the minimum of 2.0 required by Blender).
My CUDA PrefPane states that I'm on the latest possible version of CUDA [7.0.52]
Blender is aware of my GPU and does not have experimental features enabled:
My 640m GPU is very weak, and should certainly render slowly. However, it fails to render anything, even the just default cube, even just the default sky with nothing else in the scene. I don't understand why this is the case.
After getting the same error a few dozen times, one instance "succeeded"–that is, it didn't throw an out of memory error!
Instead, Blender displayed this:
...and gradually brought my computer to a standstill until it rebooted itself.
Any advice or suggestions for things to try will be greatly appreciated. CPU rendering gets very tedious very quickly, especially for complex scenes.
EDIT: VRAM Usage
I'm not sure how to precisely measure VRAM usage, but iStat menus shows a rough estimate. This is what it looks like before and after turning on Blender (without rendering anything):
If 90% of my VRAM is being used by Blender window rendering + system stuff, that leaves 51.2MB free. A fresh-from-reset scene has a memory footprint of about 10MB (according to information panel). If I take the error to be accurate, that means the memory footprint of the scene in VRAM is about 5x the normal scene memory footprint, which sounds crazy, but certainly possible.