I'm currently working on a VFX scene where I add volumetric fire to parts of my scene but in my footage I have objects that move in front of the camera I modeled each object in Blender and decided that the holdout shader would work great for masking out these objects since blenders z-depth pass doesn't work for volumetrics.
The holdout shader works perfectly but the problem is the edges of the mesh with the holdout shader create a very sharp contrast between the alpha and the fire.
Since I don't think it's possible to do blur the edges of the holdout material in compositing is there a way to create a node based holdout material that smoothly fades to alpha on the edges of the mesh removing the harsh edges?
here is a simplified blend file of the issue the holdout shader would be used to mask out 3d geometry with camera tracking but the edges of the holdout where the orange flames are behind the suitcase are too sharp I could blur the alpha channel but this would also effect the blue flames in front of the holdout
heres an example with images from a project i'm working on:
the flames are volumetric materials and the holdout shader would be used to mask out 3d geometry with camera tracking but the edges of the holdout where the orange flames are behind the suitcase are too sharp I could blur the alpha channel but this would also effect the blue flames in front of the holdout