i have a collection with holdout on it. Couple objects inside of that use alpha values on Principled BSDF and I’ve noticed that when the value isn’t exactly 1 or 0, it will show up on the render as half transparent object. Is this the expected behavior or can I do anything to competely cutout these objects still? Preferably by keeping the alpha values as they are.
If you look at it closely, there is a clear correlation between the Alpha value and the appearance of the objects. It is not that they only show up semi-transparent when they are not 0 or 1, but there is a difference between 0 and 1 as well: when the Alpha is 1, the objects are holdouts - i.e., their shape is "cut out" of the image and there is transparency behind them, no matter if you have set Film > Transparent or not, even if there is an HDRI in the background, the holdout objects are cut out and there is transparency. They do appear in reflections and throw shadows though.
But that is not what happens if the Alpha is set 0. In this case, the objects are simply transparent, i.e. they are invisible in the scene and also do not appear in reflections or throw shadows or anything. However they do not cut out everything in their place.
The Holdout setting for a collection is like adding a Holdout shader node to all materials of the object. The Alpha channel in the Principled BSDF is like using a normal opaque shader and combining it with a Transparent BSDF via Mix Shader node, and the mix factor is the Alpha value.
Now the Alpha value effects the collection Holdout in a way as if the objects had a material with a Holdout nand a Transparent node mixed together, and the Alpha value controls how much the object is transparent or holdout.
As you might have noticed, setting Alpha for example to 0.5, the object becomes semi-transparent - but instead of showing half transparency and half the Principled BSDF, it shows half holdout. So the Holdout replaces the opaque part of the Principled BSDF.
Let's say I have this scene. The three cubes are together in a collection, they all are opaque with an Alpha value of 1 in their Principled BSDF and there is an HDRI in the background.
Now I enable Holdout for the collection of the cubes. Wherever they are in front of something, they mask out the image and show transparency. But they are still "visible" to the other objects in the scene, they are in the reflections and throw shadows.
If you now set all Alpha values on the cubes to 0, this would be like when you mix them with a Transparent BSDF and set the mix to 100% transparent. They are invisible to the camera and the scene objects, no reflections, no shadows. But also no masking. This is the same as if you had no Holdout enabled - because the Holdout simply spoken replaces the opaque part of the shader - since this is like 100% transparent, there is no opacity to be replaced with a holdout.
If I now set the Alpha to a value between 0 and 1 you can see, the cubes are partly transparent and show what is behind them (like normal transparency) and partly show a transparent background (like a normal holdout). Notice that the semi-transparent do not show their color (only in the reflections).
If the cubes would not be affected by the holdout and simply use their Principled BSDF settings with partial transparency, it would look like that, the cubes showing their colors although being partly see-through.
A slightly tedious workaround (depending on how many semi-transparent objects you have in the Holdout collection) would be to mask those objects with a Cryptomatte node in the Compositor. Let's re-use the example scene from before. This time, two cubes are opaque and therefore masked out completely like they are supposed to be, but the one on the right has an Alpha value < 1 and is just partly masking the image:
What you can do now is go to the View Layer Properties tab and under Passes enable Cryptomatte > Object.
This allows you to use a Cryptomatte node - not Cryptomatte (Legacy)! - in the Compositor like this: you need to have Backdrop enabled and use a Viewer node. With the rendered image in the background, click on the + button next to the Matte ID field in the Cryptomatte node. This enables an eyedropper tool with which you can click on the semi-transparent cube to select it. If you now plug the Pick output into the Viewer node, you can see that the selected cube is masked.
Of course what you want for the alpha of the cube is the exact opposite, so you can use an Invert Color node to invert the mask. If you now plug this mask and the Alpha output of the Render Layers node into a Math node set to Minimum, the semi-transparent grey value of the original alpha will be replaced with the black from the inverted Cryptomatte. The result can be plugged into a Set Alpha node and replace the original alpha channel.
The tedious part of this process is, the Cryptomatte node does not work with collections, so if you have a lot of semi-transparent objects in the Holdout collection, you have to select them all separately one by one. I should also mention, the Cryptomatte node does not work with volumetric materials (neither does Holdout).
