Unsupported CUDA version 5.0 detected, you need CUDA 6.5

Question

I've been happily using CUDA with cycles for a while now - set Cycles Renderer up top, then File->User Preferences->System->Compute Device=CUDA and in Preferences(bottom right panel)-> Render tab->Render section->Device=GPU Compute. Can then shift-z to use cycles to render the preview in addition to just rendering.

But firing it up today results in:

Unsupported CUDA version 5.0 detected, you need CUDA 6.5.

Which is strange, because:

> ls -la /usr/local/
cuda-5.0
cuda-6.0
cuda-6.5
cuda -> /usr/local/cuda-6.5

> export | grep -i cuda
declare -x CUDA_HOME="/usr/local/cuda"
declare -x LD_LIBRARY_PATH="/usr/local/cuda/lib64"
declare -x PATH="/usr/local/cuda/bin:..."

> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Thu_Jul_17_21:41:27_CDT_2014
Cuda compilation tools, release 6.5, V6.5.12

As far as I can see my environment is already set for 6.5. I get the same for both the Fedora 21 package (2.72) and one I compiled from the source (2.73). How is blender finding the older version? I suspect I've done something to mess up my environment but can't find what it is. Any ideas?

---

Update:

strace shows blender goes straight to the cuda-5.0 directory, perhaps due to this:

> grep "/usr/local/cuda-5.0/" /usr/bin/blender
Binary file /usr/bin/blender matches

Reinstalling cuda-6.5 didn't fix it, but it did break my permissions, so:

sudo chmod -R o+rX /usr/local/cuda-6.5/

I have recently updated to nvidia driver 346.35, maybe that did something to my environment. Either way it seems like blender first looks in /usr/local/cuda-5.0/bin/ for nvcc and then /usr/local/cuda/bin/ (this seems like a bug to me). If it finds cuda 6.5 first, either via simlinking cuda-5.0 to cuda-6.5 or moving/renaming the directory, I get hit with this error:

Compiling CUDA kernel ...
"/usr/local/cuda/bin/nvcc" -arch=sm_52 ...
nvcc fatal   : Value 'sm_52' is not defined for option 'gpu-architecture'

Similar blender error mentioned here and here, and theano error here. I have just moved from a gtx titan to a 980. Maybe that's related.

UPDATE: Yes, it's the new card. Going back to the titan works just fine. Surprised others don't have similar issues with the 980.

Answer 1

As your /usr/local/ directory shows you have installed multiple versions of CUDA (/usr/bin/nvidia-uninstall would only uninstall the last one). When upgrading I had first to uninstall all CUDA files:

From pre-installation-actions

Use the following command to uninstall a Toolkit runfile installation:

$ sudo /usr/local/cuda-X.Y/bin/uninstall_cuda_X.Y.pl

Use the following command to uninstall a Driver runfile installation:

$ sudo /usr/bin/nvidia-uninstall

Use the following commands to uninstall a RPM/Deb installation:

$ sudo apt-get --purge remove <package_name>          # Ubuntu
$ sudo yum remove <package_name>                      # Fedora/Redhat/CentOS
$ sudo zypper remove <package_name>                   # OpenSUSE/SLES

Check for remaining packages:

$ dpkg -l

and remove them also. After a re-install of the most recent version should work.