Recombining render layers - sky renders incorrectly, incorrect rendering on objects

Question

I am trying to apply a blur effect to a spotlight beam - my goal is to create a somewhat smudged effect, and I am trying to accomplish this by rendering the objects, spotlight beams and sky on different layers, however I am having trouble recombining these layers in compositing. The sky renders on top of the beams, or if I check the right "use alpha" toggles in the Z combine nodes, the sky doesn't render at all. The spotlight also renders behind the object it is emanating from - this is sometimes desirable, but it also happens when the spotlight is facing the camera. What I want is simple, for the spotlight beam to render in the correct order depending on how close each pixel is to the camera and for the sky to render behind everything.

I hope my goal has been made specific enough.

Answer 1

Thanks for uploading the Blend File straight away. It would have been impossible to give an answer without it.

Your depth pass looks like this:

You can enable the Mist pass instead (raise the Mist Depth in the World Properties)

For the beam, Color Mix > Add a blurred image on top of the original one:

Since your Spotlight Beam Material is a Volume, it will not produce a useful Mist Pass (seems to be only affected by surface shading). But you can 'cheat' by using an inverted, color ramped version of your blurred beam, so you can use it in the Z Combine node.

The sky background doesn't produce a useful mist pass either, so you could cheat here again and use the (heavily crunched) color ramped sky image as Mix Factor for a Mix RGB and that should do the trick.

Note: I would recommend rendering Cube and Spotlight-Mesh on different foreground/backgroud layers, maybe with tranpsarent background, so you get clean alpha channels to play with.