My two cents:
The "dots" on the scene are way to large, and you are using very few of them. With large trackers with no sharp edges (or clearly distinct features) is very likely that you are getting false positives and inaccurate tracking.
Also, the trackers are very close together, to get a proper 3D reconstruction you want the trackers to describe the space, so put them further apart and make them of a color that is clearly different than the background:
Keep in mind that to successfully reconstruct a scene blender needs some parallax information. Your camera hardly moves. You'll get a better solution if you walk a bit around the scene. Shifting perspective is the best way to estimate 3D space out of a video shot. In other words blender needs that the tracker points move at different rates to be able to estimate what is closer and what is further away.
Please take some time and go through the suggestions listed on this post. Hopefully they will help you achieve accurate motion tracking.