Yes your topology is bad, ngons create these artefacts, try this topology instead (create a hole, extrude inwards and right click > *LoopTools > Circle*):

[![enter image description here][1]][1]


  [1]: https://i.sstatic.net/uZYw9.jpg