I need to track the camera in a very long panning shot. Because the start frame and end frame don't have common reference points, I am only able to get fragmented track points. I can detect a camera pattern in the graph view but when I try to build this, the average error is too high. Is it still possible to track a shot like this?
Yes, this is possible, although you will need to use the tripod solver. Also, you will have a difficult time doing a 3d scene reconstruction, because blender can't track depth in a panning-only shot. Also, make sure there is no zooming, as blender doesn't handle that correctly as of yet. Finally, make sure you have a decent number of solid keyframes (10ish) at each frame, and that the trackers are getting disabled AS SOON AS they begin to deviate (or be "obscured" by the edge of the screen).
Here are is the "Ultimate" BSE answer on motion tracking (required reading for anyone interested in blender's motion tracker):
Here is an AgenZasBrothers overview:
When the markers move out of the visible frame you should disable them so they aren't considered when solving. Inversely you can also have markers disabled before they come into view and enable them only while they are visible.
You can find the enabled property in the marker properties panel P, when disabled a marker is shown as a plain box without the resize handles or centre point. Like other tracking properties, this is automatically keyframed as you change it.
You can also use box select B to select all the markers that are off screen and use ⇧ ShiftD to disable the selected markers.
The same can be done in the clip editors graph view by selecting the keyframes you want to disable and pressing ⇧ ShiftD.
There is also a margin setting that can automatically disable the marker as it reaches the edge of the image, this can be set for selected markers as well as the new marker settings.