Why doesn't the normal map hide the inaccuracies of the low poly model with smooth shading?

Question

I'm trying to bake a normal map from a high poly to a low poly model. If I bake on low poly with smooth shading I get horrible artifacts :( But if I bake with auto-smoothing then I can see the sharp edges on low poly. Give me some advice, please. I think I need to add chamfers, but I would like to avoid extra polygons low poly mesh high poly mesh] normal map on low poly mesh with smooth shading normal map on low poly mesh with auto-smooth

Answer 1

The most fundamental rule of normal map baking is, however the normals are at the time of the bake, you never change them after that. The normal map represents vectors relative to your existing normals. If you bake then change, those vectors no longer make sense.

Then, you should never, ever use autosmooth on a normal mapped mesh-- which means, you don't bake with autosmooth on either. The reason for this is that autosmooth changes the normals depending on the face angle. If you change the normals-- again, your baked normal map won't make any sense.

Generally, you should use smooth shading on your low poly (for both baking and rendering.) Let's look at what happens when you don't:

Now, that does mean that you have to be careful about your smooth shading. Because remember, normals don't interpolate along quad surfaces; they interpolate along triangles. When we take this:

our long skinny quad turns into two long skinny triangles. And the normals-- thus the rays-- don't interpolate along the face the way that we'd expect. There's one vertex normal missing from the quad for each triangle, and that biases the triangle toward one angle or another.

That's, generally speaking, okay, because like I said, the normal map is relative to our existing normals, so it will counteract the effects of triangulation. Except, as the high poly surfaces gets further from the low poly surface, that triangulation gets baked into where the ray eventually strikes the high poly. So, you keep the high poly as close to the low poly as you can, because at a distance of 0, that difference in ray angle doesn't mean anything. If you can't keep them close, then you need to be careful about these long skinny quads on your low poly (try to keep your quads square.)

So: bake and use with smooth shading; keep low poly as close to the surface of the high poly as possible (including, avoiding ray extrusion.)