Can I use a vertex shader and Cycles at the same time?

Question

For my latest project, I to radically alter the vertices of my Mesh object depending on where they are on the scene. For this, I wanted to the the linear algebra and use a shader, with something like gl_Position = gl_ModelViewProjectionMatrix...

Sadly, I can't figure out how to use one of these while simultaneously having a cycles shader present (am I correct in saying cycles is only a color shader, ie, you can't use it to double the size of an object?)

A python script would be good. Shape keys would be good, but they would have to be able to interpolate such that they are applied to ONLY part of the mesh at a time! Which leaves me with vertex shaders.

Thanks in advance. (PS: I'm trying to project part of the scene as 2D and part of the scene as 3D, if someone has a wildly different approach!)

Answer 1

Cycles uses a very different rendering system and is independent of openGL, so features are going to be different and you won't be able to combine these shaders. There is an experimental real displacement feature that might help, but I think you would be better trying python if you have some idea what the code should look like.

You could use an application handler on the frame_change_pre event to create this effect. (https://www.blender.org/api/blender_python_api_2_77_1/bpy.app.handlers.html?highlight=handler)

If the changes are fairly simple, a bunch of shapekeys for various parts of the mesh may be easier, and just key/animate them manually.

Answer 2

While you can't use gl shaders to alter the cycles render, cycles does include support for OSL scripts which gives you the ability to create custom nodes.

This answer includes links to some examples of what can be done and this older answer also shows some examples.