And the result looks like this (the problem is not only at the red arrows, but also on other parts of the bake):
The same also happens of i bake normal maps. Normals, UVs, scaling is all OK. I never encountered this problem before and didn't find a solution on the internet.
EDIT: If i apply an Edge Split Modifier with 0% (all edges) and apply it, the bake is no longer distorted. However, i can't use this modifier.