Holdout plus glass shader problem!

Question

In my scene, I needed to hide a group of elements from the camera, so I hid them behind a simple plane with a holdout shader and as expected, the objects were invisible to the camera.

But here is the interesting part, later the client asked for some modifications and one of them required the scene to be seen behind a glass. So naturally I added the glass element - it was an automatic door - and to my surprise found a strange black square out of nowhere. Analyzing my scene I finally found that the problem was in the combination of the holdout material and the glass shader.

So the holdout shader works fine when seen directly throw the camera, but if I add a glass element between the camera and the holdout, then the holdout element becomes visible as black to the camera.

I tried switching off the transmission property on the cycles settings of the object but the result was that the holdout element became invisible to the camera BUT it stops hiding the elements behind it.

I finally resolved it in practical or artistic ways, like moving the objects away from the field of view or just changing the perspective of the camera. But the problem remained.

To clarify I attach some pics and a blend file to test the effect. BLEND FILE: you will see two views of camera. Switch one of them to renderview and play in the outliner, hiding or showing the elements to see what I mean.

Apparently I can't post more than two links so I had to put everything into a RAR to share only one link.

To clarify let's see what you will find on the pics:

1. We have a simple cube: H01.JPG

2. We want to hide it so we add a plane with holdout material: H02.JPG

3. The cube becomes invisible to the camera. Now let's add a glass object in front of them to see what happen: H03.JPG

Houston! Now the holdout element, which was supposed to be invisible is visible to the camera and looks black as the night!

4. Let's try to turn off the transmission property of the holdout element to see what happen...: H04.JPG

As you can see the holdout element becomes invisible to the camera through the glass but is not a holdout element anymore! Now we can see the cube...!!!

IMAGES + BLENDFILE:

https://we.tl/GVfLvxqWPg

Any idea is welcome!

Answer 1

The Holdout shader is a non-realistic shader that acts much like a black hole. From a data perspective, it really is exactly that.

The glass shader receives information (in the form of ray casts) about what's behind it, and what can be reflected in it. If that information is coming from a holdout shader, it will be null/black/empty/World/Sky. A glass shader that tries to refract, reflect, or transmit a holdout shader will produce some strange results.

For this reason, a holdout shader is really only useful for compositing.

If you want to have objects in your scene that don't render, there are a number of better ways to do this.

For one, you can disable "renderable" for each object in the outliner by toggling the camera button. This doesn't render the object at all in any regard. I think it still participates in simulations, but it won't be visible, nor cast shadows, nor participate in any other render calculations. This will work with both render engines:

Another option (specific to Cycles) is to turn off visibility for specific types of rays (Properties panel -> Cycles Settings -> Ray Visibility). This can be a really fun thing to play with. For example, if you uncheck only "Camera" then you will see the object's shadow, it will appear in reflections, and it will appear through glass, but it won't be visible if it's directly viewed by the camera.

A third (somewhat more complicated) option is Render Layers. You can put those objects on another layer, and not render that layer. Note: there are some constraints, modifiers, and simulations that don't like having their participants on different layers.