I am trying to bake the normals of my smooth-edged, high poly cube onto a 6 edged simple cube, but the results are kind of weird, not what I wanted. Here are the values I bake with:
The cube I want to bake:
The cube I want to bake onto:
And the results right now:
What am I doing wrong here and what is the correct way of doing it?
I don't think you haven't done anything wrong, it's just that the normal map you're using on your low-poly won't change the geometry of your low-poly, and if your low-poly have orthogonal angles they will continue to appear orthogonal. A normal map will just add some fake 3D bumps on a surface but it's not able to change the surface itself, it can't round something that is orthogonal.
I think that the issue you have here is that you're trying to bake to a sharp edged cube, which causes problems and is unnecessary. Your rays are projected along your face's normals. If those normals all point the exact same direction, some of them will miss the cube, and some others will move forward, then backwards, then forwards along the cube. Combined, we end up with some bad banding at the edges of our low poly once normal mapped.
With sharp edges, we're just using our face normals. Notice what happens as we reach the edge of the cube: our rays start to hit our bevel, extending out to up to 90 degrees-- and then, where our low poly extends past the borders of the high poly, we don't get anything. Then we move to the next face, and we're actually shooting our rays hitting a spot behind where we just were, discounting the misses.
Whereas on the smooth, our surface normals are interpolated. One side is smoothly connected to the other, so our rays smoothly move across the high poly, and we don't have any discontinuities.
So the answer here is, don't try to bake to a sharp edged cube. You're going to be overwriting its normals with your normal map anyways, so the fact that it's sharp doesn't mean anything useful, it's only screwing up your normal map bake.
What if there was some reason we really, really needed this to be a flat shaded cube? Is there anything we could do? I can't imagine why we'd need that to be the case, but yes, we could. Object space normal maps don't care about the pre-mapped normals. So we could smooth shade the low poly, bake an object space normal map selected-to-active, and then change the low poly to flat shaded, and bake a tangent space normal map for it (not selected to active, just from its own object-space normals.) But that seems like some weird stuff to do for no benefit.
