I tried to recalculate the normals and I also checked if the top vertices are on the same plane and they are. Am I modeling it wrong?
It seems I fixed it. I am still learning so I am not sure what fixed it but I: 1. applied the modifiers 2. removed UV Maps on all meshes on Object Data 3. exported to Unity 4. checked Generate Lightmap UVs box in asset imported in Unity
I think it was an issue with UV Maps.
Sometimes you get this issue with orphaned vertices, which can be fixed with Merge By Distance option. And sometimes it needs to be UV reunwrapped, as I think you have discovered.