It is my understanding that Cycles is a ray/path tracing render engine, and that it works by firing out "sightlines" into the scene to render, which interact with materials they hit in a defined manner to produce the color of a given pixel.
I know there are materials such as glass, which refract the ray so it travels at a slightly different angle, mirrors, which reflect the ray into another part of the scene, and diffusive materials which scatter several rays in different directions. Additionally, there are "special" materials like the holdout which simply force the given pixel to be transparent.
Given this information, I would assume that it is possible for there to be materials or shaders which instruct an incoming ray to behave in specially defined ways, such as reflecting directly back at the camera, or translating a set distance through the scene.
This got me wondering if it is possible for a custom shader to define such actions? I have seen OSL shaders which give materials a "toony" look, and other such styles that would not be possible using the standard nodes, but I haven't been able to find any information regarding what a custom shader can and can't do, and how it would be done.
Is OSL the only way to define custom shaders? Can a plugin be written such that it responds when any given ray hits it, or are there only ways to specify general rules for handling a ray?