One approach to finding if a shading point P is inside another object in OSL includes firing a probe ray back towards the camera from P, and detecting, whether the first face encountered faces towards P, or away from it.

This involves finding the normal of the first hit face. According to the OSL specification, I would expect this to be expressed in "common" space, which in the Blender/Cycles implementation, is world space.

This shader, tN_anomaly, is essentially for debugging - it prints formatted information about P, the probe ray fired from it, and the point hit by the ray.

#include "stdosl.h"

// DEBUG HELPER - return 1 with probability prob
int pRnd(float prob){ 

    int iP = int((noise("usimplex",P))*  1000000000) ; 
    int iI = int((noise("usimplex",I))*  1000000000) ;
    return( ((iP^iI ) % 1000000 ) < (1000000 * prob));


// DEBUG HELPER - return string repr of raytype(s)
string dbgRayTypes(){

    string sRT = "";

    if (raytype("camera")) sRT = concat(sRT,"[CAM]");
    if (raytype("diffuse")) sRT = concat(sRT,"[DIF]");
    if (raytype("glossy")) sRT = concat(sRT,"[GLS]");
    if (raytype("reflection")) sRT = concat(sRT,"[RFL]");
    if (raytype("refraction")) sRT = concat(sRT,"[RFR]");
    if (raytype("shadow")) sRT = concat(sRT,"[SHD]");
    if (sRT == "") sRT == "[NUL]";

    return sRT;


shader tN_anomaly(      
       output int hitBackfacing = 0
// fire a ray back towards the eye from the shading point
    if (trace(P,I)){


        // Built in to OSL ...
        // All following points and vectors in world space...
        // global -  P:  shading point of current object. 
        // global -  I:  vector from P to eye.
        // global -  N:  shading normal at shading point.
        // global -  Ng: true normal at shading point.

        // create containers for traced object´s values...

        string tObj; // name of object hit by the probe ray
        vector tI;   // vector from the probe ray´s hitpoint to P
        point  tP;   // point on target hit by the probe ray
        normal tN;   // shading normal at tP
        normal tNg;  // true normal at tP

        // Assign the corresponding values...

        getmessage ("trace", "geom:name", tObj);
        getmessage ("trace", "I", tI);
        getmessage ("trace", "P", tP);
        getmessage ("trace", "N", tN);
        getmessage ("trace", "Ng", tNg);
        // END SETUP

        // CALCULATE OUTPUT: Whether the hit mesh´s hit face was backfacing.
        //   If the normal of the first face hit is frontfacing to the camera, 
        //   (down I), it´s backfacing to us.

        if( dot(tN,I) > 0) hitBackfacing = 1;

        //PRINT DEBUG INFO for one in a thousand 'hit' rays
        // Output style:
        // | ray type: [RTP]
        // +----------------------+---------------------
        // |           self       |        hit: tObj
        // | I    Ix ,  Iy ,  Iy  |  tIx , tIy , tIz
        // | P    Px ,  Py ,  Pz  |  tPx , tPy , tPz
        // | N    Nx ,  Ny ,  Nz  |  tNx , tNy , tNz
        // | Ng   Ngx,  Ngy,  Ngz |  tNgx, tNgy, tNgz
        // | bf         ?bf       |        t?bf
        // Where:
        // [RTP] stands for the raytype flags produced by dbgRayTypes()
        // Ix (and likewise) is the x component of I (etc.)
        // tIx (and likewise) is the x component of the hit object´s I (etc.)
        //  ?bf is, whether P is viewed from the back, using backfacing()
        // t?bf is the output of the script, stored in hitBackfacing

        if (pRnd(0.001)){

            string dbg = format    ("\n\n| ray type: %5s\n", dbgRayTypes());
            dbg = concat(dbg, "+----------------------+---------------------\n");
            dbg = concat(dbg, format("| %14s%8s%13s%s","self","|","hit: ",tObj));
            dbg = concat(dbg, "\n| I");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",I[0],I[1],I[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tI[0],tI[1],tI[2]));
            dbg = concat(dbg, "\n| P");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",P[0],P[1],P[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tP[0],tP[1],tP[2]));
            dbg = concat(dbg, "\n| N");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",N[0],N[1],N[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tN[0],tN[1],tN[2]));
            dbg = concat(dbg, "\n| Ng");
            dbg = concat(dbg,format("%6.2f,%5.2f,%5.2f",Ng[0],Ng[1],Ng[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tNg[0],tNg[1],tNg[2]));
            dbg = concat(dbg, "\n| bf");
            dbg = concat(dbg,format("%12i",backfacing()));
            dbg = concat(dbg, "       | ");
            dbg = concat(dbg, format("%11i", hitBackfacing));

            printf ("%s\n\n",dbg);
        //END PRINT

In the illustrations, there are two unit cubes. The yellow cube is 0-1 in world X,Y and Z. It uses the shader in this node tree:

enter image description here

The hope is,that if the first face hit when probing back to the eye is backfacing with respect to the probe, the yellow cube will be shaded blue.

The pink cube's material is a simple tinted transparent BSDF. It's called 'TGT'.

OK, here's the first case. the camera is looking down Z. The pink cube is above the yellow cube, and clear of it, so you would expect the first normal to be encountered by the probe to be (0,0,-1).

enter image description here

A typical line printed by tN_anomaly is as follows:

| ray type: [CAM]
|           self       |        hit: TGT
| I  -0.00,-0.00, 1.00 |  0.00, 0.00,-1.00
| P   0.96, 0.94, 1.00 |  0.96, 0.94, 1.15
| N   0.00, 0.00, 1.00 |  0.00, 0.00,-1.00
| Ng  0.00, 0.00, 1.00 |  0.00, 0.00,-1.00
| bf           0       |           0

So far, so good. The first face of the hit object, (TGT's P, second column) is in the right place, just above Z=1, and its normal (N or Ng, second column - these won't differ with flat shading) shows as (0,0,-1). The output (bf, second column) is thus as it ought to be as well.

Now the second case: this time the cubes overlap, so we would expect the first normal encountered to be (0,0,1).

enter image description here

The typical output is as follows:

| ray type: [CAM]
|           self       |        hit: TGT
| I  -0.00,-0.00, 1.00 |  0.00, 0.00,-1.00
| P   0.96, 0.94, 1.00 |  0.96, 0.94, 1.80
| N   0.00, 0.00, 1.00 | -0.00,-0.00,-1.00
| Ng  0.00, 0.00, 1.00 | -0.00,-0.00,-1.00
| bf           0       |           0

Everything checks out: you can tell the right face of the pink cube is being hit (by looking at TGT's P in the second column). But TGT's normal (N or Ng, second column), is pointing the wrong way. And the output backfacing flag (bf, second column), which should now be 1, thus hasn't changed either, since it depends on the direction of N.

Can anyone on BSE give an innocent explanation .. or .. If this is a bug, where should I post it / ask further about it?

  • $\begingroup$ That's quite a lot of effort, to get to the bottom of this. Thanks! I'll have to try it out myself to see it work. Where would the output be printed? I'm guessing into Blender's console? $\endgroup$
    – kram1032
    Jun 3, 2018 at 19:46
  • 1
    $\begingroup$ @kram1032 The system console .. (Window>System Console) .. careful not to close the window, you'll close Blender. The format that results from the code is not quite as tidy as the one shown here, it's single long lines, not pairs of short lines. Easy enough to fix. keep the relevant 'hit' rendered area small ( a small number of pixels) to avoid printing too many lines, You can adjust the quantity by zooming in and out. $\endgroup$
    – Robin Betts
    Jun 3, 2018 at 20:00
  • $\begingroup$ @kram1032 Thank you for your edits .. part of the reason for my choice of formatting was to make it easier for readers of this question to see there wasn't a typo by tabulating the format parameters in the code - but your output is clearer. It's important to include Ng. The minimal case for exposing this anomaly is a pair of single faces, flipping the normals to test - (the same problem arises). Are you sure bf is provided by OSL? I can't find it in the spec, but you've managed to print it without defining it, so it must be coming from somewhere? With no docs, bf to what, I wonder? $\endgroup$
    – Robin Betts
    Jun 4, 2018 at 7:28
  • $\begingroup$ I thought I read it somewhere, but I could be mistaken about bf. It definitely gives weird outputs. If that's because it's, in fact, undefined, and it behaves sort of randomly, then that's entirely my bad. It'd be easy enough to remove just those lines from the script though. And a traced backfacing check is simple enough to do regardless, as you demonstrated already with the original script. Perhaps it should just do it that way (showing you what output the script actually gave for that hit in the end) - I noticed, that certain view angles WILL make the output be 1. $\endgroup$
    – kram1032
    Jun 4, 2018 at 10:17
  • $\begingroup$ Well, you haven't even declared 'bf'. So the compiler would definitely object if someone hadn't given it to you... I would leave it in.. it might help figure out at what level the error is occurring. I suspect the weird result when using 'int' will be because the messaging system is set up to use fixed blocks of floating-point memory for speed and economy, and casts everything to 'float'. I'm very interested in your exceptional cases .. I don't have time at the moment to try and reproduce them but perhaps, some time, you can put up a Blend file which shows them. $\endgroup$
    – Robin Betts
    Jun 4, 2018 at 10:35

1 Answer 1


My understanding of the Normal in OSL (and, in fact, standard cycles shader nodes) is that it always represents the normal on the same side as the incident ray. This provides for a consistency between ray interactions with a surface without having to worry about how the 'geometry normal' is oriented.

In order to determine which face (inside or outside) the ray has 'hit' you need to use the 'backfacing' property. In Cycles nodes this would be provided by the Geometry node but in OSL you can use the 'backfacing()' function.

For example :

shader backfacing_shader(
    output color Col = 0

    if (backfacing())
        Col = color(1,0,0);   // Red on the inside
        Col = color(0,1,0);   // Green on the outside

This means that to get your desired behaviour you need to check the backfacing() result and flip the direction of your normal if this returns a non-zero value.

  • 1
    $\begingroup$ Ah, I see what you mean too .. that's why I deleted the original comment. N and Ng always return a vector pointing towards the ray origin, in whatever context, so they can't be used to discriminate between front and back facing faces. At least now we know that's policy, not bug, so thanks. $\endgroup$
    – Robin Betts
    Jun 4, 2018 at 18:38
  • 1
    $\begingroup$ @Robin :: Does it really work for you? It doesn't for me. Are you sure that backfacing() is information from the trace() function call, and not from P? $\endgroup$ Jul 1, 2020 at 9:21
  • 1
    $\begingroup$ @cppBeginner :: (love the cpp double-colon) ... this is going back some way for me, so would involve reconstructing some lost .blends to be sure.. my approach would be to set up minimal test-cases.. maybe if you do that, you could post a contrary example as another question? I'll have another look, when I can. $\endgroup$
    – Robin Betts
    Jul 1, 2020 at 9:37
  • 1
    $\begingroup$ @Robin :: blender.stackexchange.com/questions/184868/… , thanks. $\endgroup$ Jul 1, 2020 at 10:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .