I often have problem with CUDA memory problem when rendering on GPU. I am using GTX 1080 Ti GPU (11GB VRAM). Is there a way to store some information (frame buffer, maybe textures) on system RAM in order to save GPU memory? I envy Octane Render users for having such possibility.
No, Cycles (version 2.79) is not able to use system RAM, all has to fit into VRAM.
These are the only ways that I know of to save memory when rendering: How can I save memory in cycles?
A lot of it can be saved by optimizing your scene and textures. That takes time or planning from the start.
If your time is money or your project just too big you might want to look into Octane or Redshift. Both offer using gpu only for computations and system RAM for storing large scenes.