Wrong looking normal map \ Missing Custom Split Normals on Export

Question

I spent last 2 full days trying to figure that out, and no luck.

I have low poly and high poly version of my model. Low poly is triangulated beacuse i want to texture it in Substance Painter. I baked normal map in Blender and it is working perfectly fine inside both eevee and cycles (1st picture)

The problem occurs when importing my low poly to substance painter or sketchfab, or I imagine to any other software. Importing model in those programs results in (if i understand the issue correctly) loosing custom split normal data or autosmooth and the normal map looks off (2nd picture).

Tried every solution i could find, the most obvious being:

-Trying countless times diffrent export options, FBX, OBJ, Smooth Normals Only \ Face. Smoothing Face only (option while exporting) seems to work best but still it has issues.

-Trying edge split instead of custom normal data, semi worked but normal map had awfull black seams (even in Blender) and i couldnt get rid of them (cage didn't help)

-Baking in Substance instead of Blender

-I'm using OpenGL format on import in substance(so issue is not coming from choosing DirectX)

The reason i suspect "custom normal data" to be an issue is beacuse the bad looking normal map can be recreated in blender when deleting Custom Split Normals Data and turning off autosmooth, or just turning off autosmooth (however i checked and substance painter seems to acknowledge autsmooth so its most likely just Custom Split Normals Data issue). But maybe i'm wrong and the problem lies somewhere else? (3rd picture)

Possibly important note: somewhere in the process of modeling i applied Weighted Normals modifier.

Sending normal map.png and blend file with Low poly and high poly(hidden in scene). Triangulate modifier is not applied just in case someone would want to check if thats the problem or something similar (but of course I was searching for solutions with triangulation both applied and not applied (other programs do it either way).

https://files.fm/u/wbthvstaq (link to .zip with Normal map and Blender file)

I'm completly out of ideas how to fix it. Could you guys help me?

Answer 1

If you're using a normal map anyways, you don't need custom normals. They can be baked into the normal map.

Duplicate your mesh. Delete custom normals on the original; make sure it isn't using autosmooth, no sharp edges. Make sure you're using your normal map on the duplicate's material, then bake normals from the duplicate to the original with a miniscule extrusion (like 0.00001 for example.) Your custom normals are now baked into your normal map, which is appropriate for the mesh without any custom normals.

Edit after further investigation: When I look at your file, the first thing I notice is that you're doing a live triangulation with "keep normals"-- which will create custom normals. If you need to triangulate a model, you need to do it before baking, and of course if you want to discard custom normals, you have to do that after applying any modifiers, like your triangulate, that create custom normals.

The other thing with the triangulate modifier is that the method you've picked-- "clip" for ngons-- creates zero area faces in your mesh. I'd recommend "beauty" at least for this mesh. "Fixed" and "fixed alternate" are good methods for live triangulation of deforming geometry, but if you're applying the modifier, I'd recommend "shortest diagonal" method for the quad triangulation.

Now, that's not to say that everything works when I do that. There are still the artifacts you're showing in the link you give in the comment below. This is a learning experience for me, because I always thought it was the "active object" normals that determined the ray's directions, but from what I can see from your mesh, it is the "secondary selection" normals that are determining the ray directions. And let's look at your custom normals for some of these troublesome faces:

These custom normals are sometimes actually facing opposite the true normal (and opposite the other custom normals.) I have a strong feeling this is the source of your problems with baking, and the source of the problems you're having with your mesh. Normals aren't supposed to do that.

I don't have a good solution for you. I'm only offering this because I think it's where you should be looking to solve your problems.

And thanks for making me take a closer look, because it is situations like this one that helps me learn more.

Answer 2

Ok, i tested presented ideas and the one given by Gordon seems to work properly, but with a little twist. Now I'm setting Low poly model without custom normals, autosmooth or weighted normals modifier. Turns out problem occurs when High poly object has weighted normals. Weighted normals produce those artifacts (visible only in other softwares), no matter if im using them as modifier or when i turn them into custom split data before export(on high poly object), and they dont even help with baking, they seem to not work (they work only on high poly itself). Normals bake "almost" properly now(they lack weighted normals effect, so model looks little bit off) but at least they are consistent between Blender and Substance Painter and without black seams and artifacts(also lowered baking margin from 16px to 1px, it had positive outcome in few cases). I have no idea why does it work that way (and it doesnt really explain much), but it is usable i think. If anyone have other ideas i will gladly check them, beacuse that method didnt really gave clean results either.