Here's how the stabilzation's supposed to work in Blender:
The trackers included in the 2D Stabilzation tab get pinned down.
The image then gets repositioned to keep those tracked points in the same place, effectively countering whatever move you had made.
Blender has no way to discern which movement is intentional and which isn't...
A workaround for stationary shots with panning (or tilting) is recreating the move after you have stablized the shot.
To do that you need first to unclick the 'Autoscale' button in the Movie Clip Editor. The repositioning of the clip is done on the compositor adding a 'Translate' or 'Transform' node. Note that you will need to rescale the image a bit.
In this case you're panning from left to right, but Blender is moving your image in the opposite direction. So you need to move it back again on the x axis from left to right, so that your picture stays in the frame.
In the Compositor add a 'Movie Clip' node with with the original video footage (or image sequence) and you plug it to a 'Stablize 2D' node.
On the 'Stablize 2D' node select the tracked Movie clip as source, so Blender can use the tracker information associated to the clip. The output of this node will be the stabilzed image.
Add a 'Transform' Node, and that's where you'll recreate your camera move.
On the Timeline go to the first frame of the clip.
Back on the Compositor change the values on the 'Transform' node to reposition your image so that it fills the frame. Add a key frame on the 'Transform' node by pressing I.
Scroll down the Timeline to the end frame, reposition the image again using the 'Transform' node and set another key frame.
You might need to play with the interpolation curves, or add a couple more key frames. If your image doesn't fill the frame correctly, showing the edges of the original image, then make changes on the scale. After some trial and error you should have a nice panning shot with no jitter.
It's possible that the lens distortion become more obvious. Undistorting might help for that.
Also, since you are rescaling the image, know there will be some quality loss (your image might be softer). The only way to prevent that is to shoot your original video in a format that is larger than your output...
Another strange effect that might happen, is that the motion blur that is already part of the original image looks completely wrong. In that case you might want to have the stabilization be "less perfect" by playing with the "location influence" values on the '2D Stabilization' tab, to let the image wiggle a bit.
Blender doesn't have a one button solution like the "warp stablize" found on Adobe's "for rent" software.