I'd like to use OSL to build a shader that gives a conditional (for mixing materials and such) on whether an object is inside another object and has a lower priority.

For instance, if I were to mix an arbitrary material with a transparent shader, I could make intersecting geometry disappear.

My end goal would be for the behavior to be (more or less) identical to using a boolean to cut out the intersecting region.

This could be useful, for instance, with complicated fluid setups, where several transparent objects are inside each other. (Say, a glass of water)

The Boolean modifier doesn't always work out, and in case of animation, it's not always feasible to manually clean up that modifier's results.

I have a script that certainly does... Something along those lines. But it has a few bugs. Here's the code, explaining my problems:

    In case two meshes intersect, render only the one, that has a higher priority.
    Effectively, I want meshes to appear, as if the one with higher priority carves 
    out a piece of the one with the lower priority like using a Bool modifier.

    1) For unknown reasons, priority checking seems to not work correctly. 
        It only even seems to do something with negative priorities, and then 
        all that matters is, that said priorities are negative. 
        I tried out other operations with mixed results. 
    2) I tried I, N, -I, and -N for the trace direction. 
        They all give different behavior, and I´m not sure what the right values are.

shader Priority(
    int mypriority = 0, 
    // An object´s priority

    output int HasLowerPriority = 0 
    // Whether it has lower priority, than the object it intersects
    int priority = mypriority; 
    // I did this to see if somehow mypriority isn't updated correctly. 
    //It didn't work.

    setmessage("priority", priority); 
    // if hit by a ray, this message can be sent.
    int DoTrace = trace(P, I); 
    // Shoot a ray. (Try out -I, N, and -N too). 
    // They all give different behaviors, none of which appear to be correct.
    // It's supposed to check whether the material the ray came from 
    // has a higher priority than the current material, and whether 
    // it hit that other mesh on a front or back face. 
    // If it's back facing, it has come from inside the other object.

    if (DoTrace) // If an object was hit
        // Get that object`s priority and orientation.
        int otherpriority = 0;
        otherpriority = getmessage("trace", "priority", otherpriority);
        int otherorientation = 0;
        otherorientation = getattribute("trace", backfacing());

        // Then check accordingly
        // Instead of "<", try other comparisions to see how the behavior changes. 
        // To me, that behavior seems completely unexpected, 
        // with the outcome only depending on the own meshes priority 
        // sign and not on otherpriority. As if otherpriority is always 0, 
        // suggesting there is some error there. But the script compiles just fine.
        if (priority < otherpriority && !otherorientation)
            HasLowerPriority = 1;
        //else // Pretty sure this is unnecessary too. I only added this for debug.
          //  {
            //HasLowerPriority = 0;

And here is an example blend file. Try changing the two objects' materials and the mentioned details in the script.

Also, for comparison, try activating the boolean modifier on the cube so see how it ought to look.
More or less: The Boolean modifier actually introduces what appears to be some Z-fighting. Ideally, this shader would also circumvent that.

  • $\begingroup$ Its not quite the same, but This answer, and the commentary,may be helpful $\endgroup$ – Robin Betts May 29 '18 at 13:25
  • $\begingroup$ Thanks! Weirdly, the Dipper shader doesn't seem to work as advertised. In a very similar setup, switching between Diffuse shaders, I don't seem to get any effect at all. (Only one shader is shown) Trying, instead, transmissive ones, I do see an effect, but it's weird. The "dipped" color isn't fully replaced. It's weirdly mixed. Given how you only ever generate 0 and 1 as outputs, I am clueless as to why. The also linked "Multiple transmissive objects as just one surface" by Secrop, if not quite as close a match, seems to work perfectly though. I'll see if I can't adapt that. $\endgroup$ – kram1032 May 29 '18 at 19:12
  • $\begingroup$ Unrelatedly, I noticed that my script has large portions that effectively do nothing. I'm not entirely sure why. But a far simpler script will give the exact same outcome. - Meanwhile, both your and Secrop's scripts can apparently be simplified quite a lot while still yielding the exact same outcome. $\endgroup$ – kram1032 May 29 '18 at 19:16
  • $\begingroup$ It would already help a lot if I were able to inspect values without having to use the shader in an actual material. I'd love a "Viewer" node in the Material node editor that takes any socket as input and produces a reasonable output accordingly. A number? Show me the number. A string? Show me the string. An image? Show me the image. A closure? Show me a material preview. That would be so crazy useful for debugging in general. But also for custom OSL shaders. $\endgroup$ – kram1032 May 29 '18 at 19:40

This is a partial solution. For simple scenes it'll work fine, but more complex setups will run into problems very quickly.

 * Checks if, for this ray, the object is already inside the named object.

    shader Intersects(   
        string Container = "",
        output int Intersects = 0
        if(trace (P, backfacing() ? -N : N )) {
            string name;
            getmessage("trace", "geom:name", name);
            if (name == Container) {
                Intersects = 1;

This shader returns 0 if there is an intersection with the named object, and 1 if there isn't.

It is adapted from a script by Secrop found here, as well as the Dipper script Robin Betts suggested to me from an earlier answer here.

I essentially just combined the two, while stripping them both down to the bare minimum. I'm not entirely sure, but I believe there is a bug with the getmessage() command. It appears to often do nothing, rendering much of the related logic unnecessary/redundant/ineffective. For instance, in Secrop's script, neither the backfacing() check, nor the material:index check(!) appear to do anything at all.
As far as I can tell, there are no such redundancies in this script.
That being said, it's entirely possible, I'm missing something here. Certainly, adapted from Robin's script, the call getmessage("trace", "geom:name", name); works just fine, so I don't know where the issue lies.


Here is an example setup.


Render and Materials

Note, the deformed cutting cube also interacts with the sphere, to close it back up. Without that, the sphere would be missing some walls and it'd render more like a shell, rather than an enclosed surface.
Currently the setup basically has the cube cut out a piece of the sphere, equivalent to the Boolean "Difference". Using different variations of this node setup, you can also get it to work like various other Boolean operations.


Same Scene from Different Perspective

The materials here are completely unchanged. As you can see, it may get confused in certain situations. Suzanne looks, like she partially is made of glass or something. Given how she's entirely made of diffuse materials, this certainly isn't supposed to happen. This must be the cube material getting confused about the location of the inside of the sphere.

Here you can see the same scene but with the roles of the cube and the sphere exchanged (just flip the mix shader input for both).

Same Scene, slightly Different Setup

Note, Suzanne does not touch the sphere. If the script worked perfectly, Suzanne's chin would be yellow throughout. It shouldn't be view-dependent, but strictly about whether a mesh is inside another.

There may be ways to fix these issues, but I'm not sure how.
At any rate, that's the reason why this won't be the accepted answer.
The reason I put it as answer anyway is because for relatively simple setups, this will likely already work perfectly.

One simple workaround, that may, however, not always be feasible (depending on your situation), is presumably to have several such masking objects, none of which, I assume, may intersect, all of which affect the meshes they intersect in a uniform way. (For instance, if the setup renders glass walls for one object it intersects, it ought to do so for all objects.)

This next one I'm not quite sure whether there is an issue here:

Cube is glass, cuts out diffuse sphere

In this variant, I made the cube, now clear glass, cut out a piece of the now diffuse sphere. To close the sphere, in the intersecting area, the cube generates a diffuse shader matching the sphere. Consequently, those faces will not be glass shader ones.

Now I'm not entirely sure. This would depend technical details. But does Cycles count this situation as escaping the glass to hit a diffuse mesh? Or will it think it was diffuse all along, making it more like the interface between air (vacuum) and this diffuse surface?

I'm not entirely sure it will actually make a difference, but I think it might: Namely in the case of total reflection. If Cycles thinks this is a glass-to-diffuse transition, the right setup, I think might cause total reflection, rendering the diffuse surface entirely invisible behind the internal reflection of the glass. If it rather considers this a case of having been diffuse all along, things will go wrong when internal reflections become important.

Unfortunately, for my ultimate use case, that would definitely be an issue. I'll have to see whether it works there.

Finally, I'm less concerned about it working with Volume shaders, but I did a few quick tests and it actually appears to work fine... To a point. It's certainly easy enough to make something look super weird.

  • 1
    $\begingroup$ As you know, I've been sniffing around in the same area of OSL recently, and come across an oddity which may explain some of the difficulty we've been having .. I've posted a new question about it. I don't hold out too much hope for finding an answer in the short-term, though. $\endgroup$ – Robin Betts Jun 3 '18 at 19:29
  • $\begingroup$ That's great! - Note, I'm fairly sure my script above actually works fine if there is just one object inside the mask mesh. There definitely is some weirdness, such that some checks appear to do nothing. But the bugs this script has, I think, stem from two objects being in the mask simultaneously, and the trace ray hitting the other object instead of the mask. That would, then, not be a Blender, Cycles, or OSL bug, but one with the current script. That being said, the mentioned strange behaviors might prevent or complicate a more sophisticated script able to deal with such edge cases. $\endgroup$ – kram1032 Jun 3 '18 at 20:00
  • 1
    $\begingroup$ There is another approach, which may help with your multiple-contained-object thing... that is to count the number of faces made of the container's material between P and a point known to be outside the container (probably the camera) .That would involve firing a ray from P, querying the material, and then firing again from the first hitpoint in the same direction, rinse and repeat, until you hit nothing. Count the number of hits of the container's material. If odd, you're inside. $\endgroup$ – Robin Betts Jun 3 '18 at 20:48
  • 1
    $\begingroup$ I just spent a little time trying to get that to work. I'll have to think more carefully about the logic. However, in the process, I got rid of one problem my script had. (I'll try to improve more before updating the answer) - View dependence. I do not want it. So, then why would I use I? - I'm now using Ng instead, and it gives me basically the same results, except consistently from all sides. I can now also see what's going on more clearly, when the ordering fails: puu.sh/Ayx8u/7104d2f936.png Note the piece of free-floating glass, and the random bits of yellow and red on Suzanne. $\endgroup$ – kram1032 Jun 3 '18 at 21:48

I've had some luck with the face-counting approach .. setting off a chain of rays to the eye-point, each starting at the last hit point plus a bit.

If the number of faces of the enclosing material hit on the way is odd, we're inside. You enter the Pass Index of the material you want to be enclosing in the materialIdx slot. Here, pink has an index of 20,and yellow has an index of 10.

There have been some intermittent fails, so I can't be sure this is stable. One time, when I removed the debugging code, it suddenly stopped working. That suggests a thread problem to me.. some sort of race condition in the renderer. I have not hammered it yet, to see what's happening.

This is the setup

enter image description here

This is the shader

shader Jordans(  
        float materialIdx = 0.0,     
 output int   isInside = 0

    point ray0 = P;       //The first of a sequence of ray origins
    float matIdx = 0.0;   //material encountered
    int   hitCount = 0;   //times enclosing material encountered
    int   traceCount = 0; //total rays in chain - limited in this version

    while ((traceCount<4) && trace(ray0,I,"mindist",0.000001)){
        getmessage("trace", "material:index", matIdx);
        getmessage("trace", "P", ray0);
        traceCount ++;
        hitCount += (materialIdx == matIdx);        

    if(hitCount%2) isInside = 1;

Yellow nodes:

enter image description here

Pink nodes:

enter image description here


enter image description here

I'm sure you can break it.

  • $\begingroup$ This is very close to what I already had yesterday. If this works without errors, I'll definitely have to figure out why mine didn't. Will test in a minute. $\endgroup$ – kram1032 Jun 4 '18 at 22:37
  • 1
    $\begingroup$ Looks like setting a minimum distance helped with some of the issues I've been having with this method, but it didn't fix it. My test scene will not entirely work this way. puu.sh/Azbo5/fb8aa85551.png as you can see, there are artifacts here. Setting a higher trace depth doesn't really help. If you want to check for these, by using N or Ng you can make the artifacts visible from any side, even if I is the right answer in the end. I think a fully working method won't have these happen regardless of the trace direction used. $\endgroup$ – kram1032 Jun 4 '18 at 23:00
  • 1
    $\begingroup$ I guess the trouble with glass is that there are bent refraction rays contributing color where you don't want it.. IOR of 1.0? Just to see? Got to go to bed now.. we should probably move any long discussions to chat.. but I don't know how that works yet. $\endgroup$ – Robin Betts Jun 4 '18 at 23:08
  • $\begingroup$ Nope. You probably answered that while I edited my previous message on you, sorry. But in fact, the glass is entirely innocent. Here is the same scene without the sphere, and with the cube set to always be entirely transparent (i.e. it doesn't even use the script.) Only Suzanne will be affected. puu.sh/AzbNT/ed25da5b62.png - These artifacts are view-dependent though: puu.sh/AzbRt/d6a82632fd.png From this side everything looks fine. Thanks for your help. Good night! If there is too much back and forth, SE will just ask you whether you want to chat. $\endgroup$ – kram1032 Jun 4 '18 at 23:17

Did a version of this in blender 2.8, had a issue with some edge weirdness, just changed mindist from 0.000001 to 0.00001 and it cleared itself. Don't know why.

shader Intersects(  
    string Container = "",
    output int isInside = 0

    point ray0 = P;       //The first of a sequence of ray origins
    int   hitCount = 0;   //times enclosing object is encountered
    int   traceCount = 0; //total rays in chain - limited in this version

    while ((traceCount<10) && trace(ray0, backfacing() ? -N : N,"mindist",0.00001)){
        string name;
        getmessage("trace", "geom:name", name);
        getmessage("trace", "P", ray0);
        traceCount ++;
        hitCount += (name == Container);        
    if(hitCount%2) isInside = 1;

  • $\begingroup$ I unfortunately can't test it for myself 'cause my computer is too ancient for 2.8 but looks good. Looks like you went with Robin's version. I assume the limitations are pretty similar to what Robin's has? Or does this work completely? $\endgroup$ – kram1032 Nov 14 '19 at 18:05
  • $\begingroup$ Same one really, just using object name instead of material. Just 0.00001 is different. $\endgroup$ – Øyvind Saltvik Nov 14 '19 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.