OSL: Anomalous normals returned by getmessage("trace")

Question

One approach to finding if a shading point P is inside another object in OSL includes firing a probe ray back towards the camera from P, and detecting, whether the first face encountered faces towards P, or away from it.

This involves finding the normal of the first hit face. According to the OSL specification, I would expect this to be expressed in "common" space, which in the Blender/Cycles implementation, is world space.

This shader, tN_anomaly, is essentially for debugging - it prints formatted information about P, the probe ray fired from it, and the point hit by the ray.

#include "stdosl.h"

// DEBUG HELPER - return 1 with probability prob
int pRnd(float prob){ 

    int iP = int((noise("usimplex",P))*  1000000000) ; 
    int iI = int((noise("usimplex",I))*  1000000000) ;
    return( ((iP^iI ) % 1000000 ) < (1000000 * prob));

}

// DEBUG HELPER - return string repr of raytype(s)
string dbgRayTypes(){

    string sRT = "";

    if (raytype("camera")) sRT = concat(sRT,"[CAM]");
    if (raytype("diffuse")) sRT = concat(sRT,"[DIF]");
    if (raytype("glossy")) sRT = concat(sRT,"[GLS]");
    if (raytype("reflection")) sRT = concat(sRT,"[RFL]");
    if (raytype("refraction")) sRT = concat(sRT,"[RFR]");
    if (raytype("shadow")) sRT = concat(sRT,"[SHD]");
    if (sRT == "") sRT == "[NUL]";

    return sRT;
}

//SHADER START

shader tN_anomaly(      
       output int hitBackfacing = 0
)
{  
// fire a ray back towards the eye from the shading point
    if (trace(P,I)){

        // DEBUG INFO DUMP SETUP:

        // Built in to OSL ...
        // All following points and vectors in world space...
        // global -  P:  shading point of current object. 
        // global -  I:  vector from P to eye.
        // global -  N:  shading normal at shading point.
        // global -  Ng: true normal at shading point.

        // create containers for traced object´s values...

        string tObj; // name of object hit by the probe ray
        vector tI;   // vector from the probe ray´s hitpoint to P
        point  tP;   // point on target hit by the probe ray
        normal tN;   // shading normal at tP
        normal tNg;  // true normal at tP

        // Assign the corresponding values...

        getmessage ("trace", "geom:name", tObj);
        getmessage ("trace", "I", tI);
        getmessage ("trace", "P", tP);
        getmessage ("trace", "N", tN);
        getmessage ("trace", "Ng", tNg);
        // END SETUP


        // CALCULATE OUTPUT: Whether the hit mesh´s hit face was backfacing.
        //   If the normal of the first face hit is frontfacing to the camera, 
        //   (down I), it´s backfacing to us.

        if( dot(tN,I) > 0) hitBackfacing = 1;


        //PRINT DEBUG INFO for one in a thousand 'hit' rays
        //
        // Output style:
        //
        // | ray type: [RTP]
        // +----------------------+---------------------
        // |           self       |        hit: tObj
        // | I    Ix ,  Iy ,  Iy  |  tIx , tIy , tIz
        // | P    Px ,  Py ,  Pz  |  tPx , tPy , tPz
        // | N    Nx ,  Ny ,  Nz  |  tNx , tNy , tNz
        // | Ng   Ngx,  Ngy,  Ngz |  tNgx, tNgy, tNgz
        // | bf         ?bf       |        t?bf
        //
        // Where:
        //
        // [RTP] stands for the raytype flags produced by dbgRayTypes()
        //
        // Ix (and likewise) is the x component of I (etc.)
        // tIx (and likewise) is the x component of the hit object´s I (etc.)
        //
        //  ?bf is, whether P is viewed from the back, using backfacing()
        // t?bf is the output of the script, stored in hitBackfacing

        if (pRnd(0.001)){

            string dbg = format    ("\n\n| ray type: %5s\n", dbgRayTypes());
            dbg = concat(dbg, "+----------------------+---------------------\n");
            dbg = concat(dbg, format("| %14s%8s%13s%s","self","|","hit: ",tObj));
            dbg = concat(dbg, "\n| I");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",I[0],I[1],I[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tI[0],tI[1],tI[2]));
            dbg = concat(dbg, "\n| P");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",P[0],P[1],P[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tP[0],tP[1],tP[2]));
            dbg = concat(dbg, "\n| N");
            dbg = concat(dbg,format("%7.2f,%5.2f,%5.2f",N[0],N[1],N[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tN[0],tN[1],tN[2]));
            dbg = concat(dbg, "\n| Ng");
            dbg = concat(dbg,format("%6.2f,%5.2f,%5.2f",Ng[0],Ng[1],Ng[2]));
            dbg = concat(dbg, " | ");
            dbg = concat(dbg, format("%5.2f,%5.2f,%5.2f",tNg[0],tNg[1],tNg[2]));
            dbg = concat(dbg, "\n| bf");
            dbg = concat(dbg,format("%12i",backfacing()));
            dbg = concat(dbg, "       | ");
            dbg = concat(dbg, format("%11i", hitBackfacing));

            printf ("%s\n\n",dbg);
            }
        //END PRINT
    }
}

In the illustrations, there are two unit cubes. The yellow cube is 0-1 in world X,Y and Z. It uses the shader in this node tree:

The hope is,that if the first face hit when probing back to the eye is backfacing with respect to the probe, the yellow cube will be shaded blue.

The pink cube's material is a simple tinted transparent BSDF. It's called 'TGT'.

OK, here's the first case. the camera is looking down Z. The pink cube is above the yellow cube, and clear of it, so you would expect the first normal to be encountered by the probe to be (0,0,-1).

A typical line printed by tN_anomaly is as follows:

| ray type: [CAM]
+----------------------+---------------------
|           self       |        hit: TGT
| I  -0.00,-0.00, 1.00 |  0.00, 0.00,-1.00
| P   0.96, 0.94, 1.00 |  0.96, 0.94, 1.15
| N   0.00, 0.00, 1.00 |  0.00, 0.00,-1.00
| Ng  0.00, 0.00, 1.00 |  0.00, 0.00,-1.00
| bf           0       |           0

So far, so good. The first face of the hit object, (TGT's P, second column) is in the right place, just above Z=1, and its normal (N or Ng, second column - these won't differ with flat shading) shows as (0,0,-1). The output (bf, second column) is thus as it ought to be as well.

Now the second case: this time the cubes overlap, so we would expect the first normal encountered to be (0,0,1).

The typical output is as follows:

| ray type: [CAM]
+----------------------+---------------------
|           self       |        hit: TGT
| I  -0.00,-0.00, 1.00 |  0.00, 0.00,-1.00
| P   0.96, 0.94, 1.00 |  0.96, 0.94, 1.80
| N   0.00, 0.00, 1.00 | -0.00,-0.00,-1.00
| Ng  0.00, 0.00, 1.00 | -0.00,-0.00,-1.00
| bf           0       |           0

Everything checks out: you can tell the right face of the pink cube is being hit (by looking at TGT's P in the second column). But TGT's normal (N or Ng, second column), is pointing the wrong way. And the output backfacing flag (bf, second column), which should now be 1, thus hasn't changed either, since it depends on the direction of N.

Can anyone on BSE give an innocent explanation .. or .. If this is a bug, where should I post it / ask further about it?

Answer 1

My understanding of the Normal in OSL (and, in fact, standard cycles shader nodes) is that it always represents the normal on the same side as the incident ray. This provides for a consistency between ray interactions with a surface without having to worry about how the 'geometry normal' is oriented.

In order to determine which face (inside or outside) the ray has 'hit' you need to use the 'backfacing' property. In Cycles nodes this would be provided by the Geometry node but in OSL you can use the 'backfacing()' function.

For example :

shader backfacing_shader(
    output color Col = 0
  ){

    if (backfacing())
    {
        Col = color(1,0,0);   // Red on the inside
    }
    else
    {
        Col = color(0,1,0);   // Green on the outside
    }
} 

This means that to get your desired behaviour you need to check the backfacing() result and flip the direction of your normal if this returns a non-zero value.