I want to achieve this outline effect outside of the game engine for animations, without using freestyle. Is it possible to make this effect work in Cycles or BI?
Yes, this is quite easy to do in cycles:
Duplicate and scale the object as shown in the video you linked.
Add a node setup using the Backfacing output of the Geometry node to mix a black diffuse shader with a transparent shader. This will go on the scaled duplicate (outline) object to provide the backface culling effect.
Disable the Ray visibility on the outline object for everything except Camera and Glossy, Transmission if you want to see the outline in reflections and refractions. This will allow light through the outline object, illuminating the main object inside.
To see the effect in the viewport:
Enable Backface Culling in 3D view > Properties panel (N) > Shading, and set the viewport color of the outline material to black:
Outlines without extra objects
Also, if you find having an extra object for the outlines rather awkward, you can generate the outline mesh procedurally using a Solidify modifier:
Add the outline material in another material slot:
Add a solidify modifier to the object you want to outline, enable Flip Normals and set the Material Index Offset to the slot where your outline material is (where the first slot is slot
0, the second is slot
1, etc.). So in this case,
Now the outline will even update in real time while you are modeling: