I tried to recalculate the normals and I also checked if the top vertices are on the same plane and they are. Am I modeling it wrong?
It seems I fixed it. I am still learning so I am not sure what fixed it but I: 1. applied the modifiers 2. removed UV Maps on all meshes on Object Data 3. exported to Unity 4. checked Generate Lightmap UVs box in asset imported in Unity
I think it was an issue with UV Maps.