Why is it overlapping?
Your polygon is not a traingle, there is very small edge (top corner) which overlaps quite easily with default inset tool, due to its relative shortness to adjacent edges.
How to prevent overlapping?
Either merge overlapped geometry after inseting or use shipped add-on called Inset Straight Skeleton (Blender 2.8x) or Inset Polygon (Blender 2.7x), which insets without overlapping. Actually, add-on automatically merges overlapped geometry on every appearance, creating extra loops.
SE: Inset Straight Skeleton enabling and usage info
Sometimes skeleton inset produces bad geometry (wavy and not even) due to precision issues with merging distance. There is a trick to obtain beter results, it involves scaling:
- Scale a polygon in edit mode n times (used n=10 factor for pic example)
- Inset with amount * n (inset should be relatively bigger)
- Scale 1/n times (everything goes back to initial size)
"Scale trick" pic example: (it is a long polygon, but in perpective view for better visibility)
