The attached animated gif shows some frames of a video with a mesh box placed at each tracking marker used to successfully track (solution error <.4) an 800 frame chunk of video.
The point is to illustrate just how poorly the camera tracker calculated: when the box is at a marker placed at the top of a faraway building on the left, it is actually visually much larger than when it is at the tracking marker closest to where the real camera was. I went to great lengths to be sure that the camera's sensor size and focal length are correct, and manually placed and followed all markers to be sure they were accurately tracking.
There is not a lot of parallax motion over the 800 frames, but there should have been enough to at least know that the point closest to the camera moved more than the point furthest away. Sometimes it seems to work, sometimes it doesn't. Does anybody know how to improve this?