The important concept here is that you track all of the elements that can give you a clear description of the geometry of the head.
Use tracking points not only the face but on the back, shoulders, etc.
When one item being tracked disappears from view, (like when you trurn around) then blender has way to guess where those points are in 3d space or reconstruct the scene or object. To fix that, you need to track other items and let blender figure out the relationship between the different tracked items, only then you'll get information for points that might have disappeared from view or that got occluded.
In this example elements for front and back of an object are tracked, note how not all elements are visible all the time, some disappear briefly and reappear:
When an tracked element disappears from view that particular tracker stops. When the same element is visible again, track it again with new tracker and join the previous tracker with the new one, so that blender knows it is the same feature being tracked and not a new one.
In other words give blender as much information on the object as you can.