The most common error is this: the shot has no parallax. Meaning that the camera is stationary, it is just rotating, but with no displacecement.
Shots like those cannot be solved with the default solver, but need to be solved as tripod. Read:
How can I get better results when doing camera motion tracking?
A few words of advise: track objects at the point where there is more contrast, like the window on the back, don't track the middle where all of the pixels have almost the same value, track a sharp edge or features that show distinct characteristics.
In this case the tracking is accurate and the tripod solution is as good as it is going to get, but know this: you will never get 3D reconstruction out of a tripod solve. You will only get a camera movement that resembles that of the original camera. Without parallax there is no way to estimate depth, meaning how far away objects are from the camera. The tracker for the window across the freeway will end up at the same distance from the camera as the trackers for objects in the foreground.
The tracking points are projected into a sphere around the camera, there is no distance information. So in this case you need to be vary careful in setting the lens parameters correctly and try to match the scale of the 3d models to that of the real world. Grab the camera and try to move it so that it is at the height of the real camera, and for the actual distance try to create some geometry that matches the dimensions of real objects. Then you might need to adjust the size of the camera to match the geometry to the objects in the scene. In other words you are up for a lot of work and lots of guessing. There are no automatic tools for accurate solving without proper depth estimation.
As an alternative, try using single image in fspy and determine the camera lens, height and scale there, and use those parameters for the tracked camera.
