when baking normals, do I have to set shading to smooth or flat for the low mesh?

Question

When I set the shading for the low-poly to flat, I get the sharp edges baked into the normal map which I don't want. When I set the shading for the low-poly to smooth, the smooth-shading gets baked into the normal map, which I also don't want.

1. How can I simply bake my high-poly details onto the low-poly mesh?
2. Why does Blender even change baking behavior when changing the shading mode of the low-poly mesh? As far as my understanding goes, this should have no effect - the goal is to bring my high-poly details to low poly. I do not care either about sharp edges of the low poly or the ugly artefacts of the smooth shading. Is there a purpose behind such behavior?

This is when baking with smooth shading for the low-poly:

The marked area should be a completely flat area, yet the smooth shading artefacts also get into the normal map.

And this is when baking with flat shading for the low-poly:

This looks better - but, alas, sharp edges of the low-poly cylinder end up in the normal map as well!

Answer 1

A tangent space normal map is a bake of normals relative to the low poly's normals. Whenever you make a tangent space normal map, it is going to take your existing normals, whether they're flat shaded or smooth shaded (or custom split normals, for that matter), and bake them into your normal map.

So I think you just have a bit of a misunderstanding of what a tangent space normal map is. You shouldn't be looking at your baked image to judge a normal map-- it can't be judged on the basis of the image alone. You should be judging it on the mesh for which it was built.

Now, it should be said, there is someplace where it's important whether your low poly is smooth shaded or flat shaded or something else. That's because whenever Blender bakes anything, it looks in the direction of the baking normals to figure out what to bake. So, for example, if you bevel a cube, and then bake to a flat-shaded cube, you'll find that some bake samples never actually hit the high poly. Unless you know really well what you're doing, you generally want a smooth-shaded low poly that completely encloses your high poly.

It's not very clear exactly why you want your normal map to somehow be independent of your low poly's normals. I think there's probably some misunderstanding of what a normal map is, or what the colors mean. But if you're really sure that you don't want your normal map to depend on your low poly's normals, you might consider baking an object-space normal map instead. Object space normal maps are stored relative to object space, rather than relative to the sample's normal (which turns into the Z axis for the tangent space.) However, object space normal maps will still be affected by your normals in that the low poly normals will still determine which way the rays shoot from the low poly (or cage) to hit the high poly. Object space normal maps are not any good for deforming geometry, which is why tangent space normal maps are seen more frequently.