Editing a mesh for a while I didn't notice that I created a n-gon. After adding a subsurf modifier the mesh heavily deformed, it took while until I found this:
Is there an easy way to select n-gons?
Yes.
Press 3D view > HEader > Select > Faces by Sides:
Then set the type to greater than and the number of vertices to 4 in the redo panel (F6 or at the bottom of the tool shelf):
Edit Blender 2.8
This option is now in Select > Select All by Trait > Faces by Sides