I have a a GeForce GTX 980 Ti and a GeForce GTX 760 installed in my system. I just tried rendering a scene that I know will render fine with the GTX 980 Ti but when I render with both cards I get "CUDA error: Out of memory in cuArrayCreate(&handle, &desc)". I know exactly what is going on. Blender lists the Compute Devices as GeForce GTX 760 + 980 Ti. Even though the 980 Ti is in PCI bus 1 and the 760 is in PCI bus 2, it is obvious that Blender is using the RAM from the weakest card. Since this seems like a real handicap, I was wondering how I can force Blender to do the right thing, the common sense thing, and use the memory from primary card with the largest amount of RAM when rendering my scene with both cards.
Blender is actually using the memory of BOTH cards. The scene has to be fully loaded into both GPUs as they are entirely separate. They cannot share memory. You are limited by the size of the smaller card. The order in the preferences makes no difference.
There is no way to fix this, and no way to have it not use selected cards with insufficient RAM to hold the scene. The only option is to set it to select only the larger card for the time being.