is there a way to blur the edges of the holdout shader

Question

I'm currently working on a VFX scene where I add volumetric fire to parts of my scene but in my footage I have objects that move in front of the camera I modeled each object in Blender and decided that the holdout shader would work great for masking out these objects since blenders z-depth pass doesn't work for volumetrics.

The holdout shader works perfectly but the problem is the edges of the mesh with the holdout shader create a very sharp contrast between the alpha and the fire.

Since I don't think it's possible to do blur the edges of the holdout material in compositing is there a way to create a node based holdout material that smoothly fades to alpha on the edges of the mesh removing the harsh edges?

here is a simplified blend file of the issue the holdout shader would be used to mask out 3d geometry with camera tracking but the edges of the holdout where the orange flames are behind the suitcase are too sharp I could blur the alpha channel but this would also effect the blue flames in front of the holdout

heres an example with images from a project i'm working on:

the flames are volumetric materials and the holdout shader would be used to mask out 3d geometry with camera tracking but the edges of the holdout where the orange flames are behind the suitcase are too sharp I could blur the alpha channel but this would also effect the blue flames in front of the holdout

Answer 1

From your screens it seems to me like you want to smooth part of an edge that is (in reality) optically smoothed by Bloom Effect. If you can’t use bloom effect in post-production directly, try this … take your mask object, blur its alpha, invert and subtract it from alpha of fire object.

There are two issues to solve ...

• ad First - your example material setup use for Volume - Emission shader, that is visible in viewport render, but invisible in render (F12). Material is rendered, but alpha for this material is not generated. From developer side the bug is viewport vs render (F12) inconsistency, but F12 is as designed = emission without alpha is correct. This can be solved by adding Volume Scatter shader.

• ad Second - in blender wasn’t a way to get ID pass (or Cryptomatte) of object with volumetric material. From 2.82 beta build, blender use AOV that can output any state of material node tree. So I used AOV to separate specific alpha for Blurring. (Properties Editor > Layers > Passes > Shader AOV, create one. Go to Shader Editor, add AOV Output node with proper name and set white colour. For more about AOV see Blender Manual or Blender Today #86.)

Note: I had to add Transparent shader to Surface (feeding only Volume didn’t object pass as well). Pass of this volumetric object get object shape, not volumetric texture.

Since your scene consist only from fire I blurred mask object (cube). To get "fire" pass you can use Alpha socket ...

Show before/after. Sorry for poor render (6 samples only).

... or convert render with ColorRamp into black&white image. That is not mathematically accurate, but gave me nicer result.

(I created also pass for orange fire, but in this comp was not needed.)