Extracting refraction info and mix

Question

I'm newbie in Blender, and hope that my question would be relatively simple :)

Now i render some images for my game by Blender, which i feed to Pixel Shader (PS) and achieve refraction effect by displacement of original texture.

Here is the PS

float4 PixelShaderDisplacementFunction(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(assembleSampler, input.UV.xy) * 2 - 1;
    float2 UV = input.Position1.xy + color.xy / 64;
    return tex2D(screenSampler, UV);
}

It uses R & G channels as XY offset position of currently rendering pixel and looks up for proper pixel of already rendered screen texture.

Just for ex. if R == 0.5 (0x80) then there will be no displacement at all, but 1 - (+64 pixels ie to the right) and (0) - (-64 pixels to the left), for G is the same, so you should get the idea.

Well, for now i can extract refraction by using such a method - 256x256 ramp texture where R & G for X & Y respectively increasing from 0 to 255 is placed as background texture for the scene, and scene itself is rendered with 256x256 resolution (note that object must have perfect white color, so there refraction just moves ramp texture color to a different place without any other changes)

After a scene is rendered, i subtract original ramp texture, and actually get the offset for each pixel.

Divide and Add nodes are used to adjust color space, ie ((SrcColor + 1) / 2, or SrcColor / 2 + 0.5) and color (0,0) becomes (0.5,0.5), but (-1.0, -1.0) -> (0,0), and (1.0, 1.0) -> (1.0, 1.0). Btw, i'm not sure if Blender texels internally are vector4 type, so this math may not work properly, but not verified it yet.

Other nodes are used to remove some artifacts and output to 64x64 image.

So, this method does work, but also i would like to use other channels - Blue for index in lookup texture (1D with 256 colors, say VGA std palette), and Alfa for opacity of this indexed color. Then the PS should be something like this

float4 PixelShaderDisplacementFunctionAdvanced(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(assembleSampler, input.UV.xy);
    float2 UV = input.Position1.xy + (color.xy * 2 - 1) / 64;
    float4 srcColor = tex2D(screenSampler, UV);
    float4 paletteColor = tex1D(vgaSampler, color.z);
    return srcColor * (1 - color.w) + paletteColor * color.w;
}

And the question is, how can i render a scene twice - 1st as now, and second with tinted object with some semi-transparent color, then subtract 1st from 2nd to get pure tint color with opacity, find most suitable palette index from supplied texture and then mix it to final image ? Not sure, but maybe somehow with Python ?

P.S. While this PS works pretty cool, but not perfect, so objects just do refract and not actually visible, seems like a ghost effect, but tinting should make this look perfect.

Thanks for your help !

Answer 1

Seems its not interested by anyone, but i'had solved the problem.

Here is the video https://youtu.be/S4yfnxffJmA

And here is preview

Actually i've compressed all the 3 channels to one (to 6x6x7 palette), but it was not looking good. Had to refactor many things so there are 4 + 4 channels, (2 for displacement + 2 are free for future use) & 4 for tint as standard RGBA.

Here is the final PS

float4 PixelShaderDisplacementFunction(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(assembleSampler, input.UV.xy);
    float2 UV = input.Position1.xy + (color.xy * 2 - 1) / ZoomMultipliedBy64;
    float4 srcColor = tex2D(screenSampler, UV);
    return srcColor;
}

ZoomMultipliedBy64 = RenderMatrix.M44 * 64f, to neutralize effect of zooming