Why does the normal map look inconsistent?

Question

Data: 3D RIGGED PEOPLE >> rp_carla_rigged_001_FBX in free-3d-people.

Setup for "rp_carla_rigged_001_yup_a.fbx": Default + Eevee + Transparent + Camera(0,-3,1)

Color:

Normal:

Normal in "Non-Color"(There are still obvious inconsistencies among different UV islands):

Seams from Islands: Code for visualization:

import cv2
import numpy as np
from vedo import *

path_img="/home/lab9/Pictures/untitled.png"
img_ori = cv2.imread(path_img, -1)
img_ori = cv2.cvtColor(img_ori, cv2.COLOR_BGR2RGB)
img_ori = img_ori[300:400, 810:850, :]

img = img_ori.copy().reshape(-1, 3)
img = img / 255.0
img = img * 2 - 1

if True:
    plt = Plotter(axes=2)
    B = Box(pos=(0, 0, 0), length=4, width=3, height=4).alpha(0.1)
    O = np.zeros_like(img)
    L = Lines(O, O+img, c="r")
    plt.show(B, L)

Result:

Question: I have a need to render a normal map as a texture.

When choosing a color map("_dif.jpg") as the texture, it looks consistent.

However, choosing the normal map("_norm.jpg") as the texture looks inconsistent, but the results of the visualization are all pointing to the positive direction of the Z axis (it should be considered correct).

What about the inconsistencies between the rendered result(as color) and the visualization result(as normal/vector)?

What transformation can I do to make the normal map look more "consistent"?

Update1: (based on Blunder's comment)

I create a plane in Blender, after sub. 100 and sculpt:

Another two planes:

without rotation: with rotation:

000 090
270 180

Baked result:

without rotation:

with rotation:

Correction code for the second UV island, rotateX=90 equals:

x_new = -y_old
y_new = x_old

import cv2
import numpy as np
from vedo import *

path_img_N001 = "N.001.png"
path_img_N002 = "N.002.png"
img_N001 = cv2.imread(path_img_N001, -1)
img_N002 = cv2.imread(path_img_N002, -1)
img_N001 = cv2.cvtColor(img_N001, cv2.COLOR_BGR2RGB)
img_N002 = cv2.cvtColor(img_N002, cv2.COLOR_BGR2RGB)

tl = (192, 576)
bl = (447, 576)
tr = (192, 831)
br = (447, 831)

img_N001 = (img_N001 / 255.0) * 2 - 1
img_N002 = (img_N002 / 255.0) * 2 - 1

ou = tl[0]
ov = tl[1]
img_N001 = img_N001[ou : ou + 255, ov : ov + 255].reshape(-1, 3)
img_N002 = img_N002[ou : ou + 255, ov : ov + 255].reshape(-1, 3)
img_N002[:, [0, 1]] = img_N002[:, [1, 0]]
img_N002[:, 0] *= -1  #!


if True:
    plt = Plotter(axes=2, shape=(1, 2), sharecam=False)
    B = Box(pos=(0, 0, 0), length=4, width=3, height=4).alpha(0.1)
    O = np.zeros_like(img_N001)
    L1 = Lines(O, O + img_N001, c="r")
    L2 = Lines(O, O + img_N002, c="g")
    plt.show((B, L1), at=0)
    plt.show((B, L2), at=1)
    plt.show(interactive=1)

We can see that after rotation correction, the result is consistent.

Maybe we can restore the relative rotation between UV islands by the following observation:

The first plane(without rotation):

The second plane(with rotation, rotX=90):

The second plane(with rotation, rotX=180):

The result obtained after top-right rotation correction (bottom not processed, still inconsistent):

Now the problem is: are there any Off-the-shelf tools to calculate the rotation parameters of UV island?

Answer 1

As Blunder offered in comments, it looks different across the seam because the UV islands are differently rotated across the seam. This is normal and expected.

A tangent space normal map is a (remapped) array of vectors in the tangent space of the sample that they modify. Tangent space is a special, per-sample space where the +Z vector points in the same direction as the surface normal and the +X vector points in the direction of increasing U (from the UV map.) So when there are seams, that +X vector is probably going to be pointing in a different world direction, so the normal map can look discontinuous even if the vectors it contains, properly interpreted in world space, are not discontinuous.

You ask, "are there any Off-the-shelf tools to calculate the rotation parameters of UV island?"

There are plenty of ways to figure out which direction the tangent is pointing. One way to do it is to plug a 1,0.5,0.5 color into the normal map, which represents a normal pointing in tangent space's +X direction. Or, we could just plug a tangent node into output, which is the same thing. Either gives us the world space vector representing the tangent, the thing which is discontinuous across your mesh that causes the normal map to be discontinuous:

We can see that the tangent is discontinuous across the seam, and that discontinuity will show up in any tangent space normal map we make for this mesh, again, even though the vectors the normal map represents won't be discontinuous.

We could bake those tangent vectors if we wanted, probably to a floating point .exr to preserve negative values and precision (or remap to the 0,1 range, like we do with normal maps, and save as a .png or something.)

But the tangent is a vector, and you want rotation. Rotation is relative-- what do you want rotation relative to? The world +X axis? That's mostly doable, but it will fall apart for faces in the world YZ plane, and we'll start to have terrible precision as faces approach that plane. It's just like this radial tangent on the left. There's a pole on the top of Suzanne's head where the answer doesn't make any sense:

And the tangent for the cube projection model, on the right, demonstrates even more discontinuities than you have, so a cube-mapped tangent isn't some natural, no-rotation choice either. The whole problem is a little bit like how to cover a sphere with hair. It can't be done without any discontinuities or poles.

There's no right answer for where the tangent should point, so there's no single reference for what is the rotation of the tangent. This is related to why we use tangent space normal maps to begin with, because we need some baseline coordinate system to measure normal maps on deforming meshes, and we need to make an arbitrary (but stable) choice about one of the basis vectors of that coordinate system. If there were some "true tangent" to which we could measure our tangent's angle, we'd use that instead.

Answer 2

Just for the sake of completeness, I just remembered that you can use the Solid shading with the Normal Matcap to display the model in normal map colors:

The Solid mode does not display any normal map data from the material. But you can render Matcaps as material as described here: Matcap materials as a material for render?

If you combine this with the model's normal map then you can render her in these normal map colors.

In the Render Settings > Color Management set View Transform to Standard. This way you will see the raw colors. It should be 8080FF (hex) for everything that is perpendicular to the camera.

Shader nodes: