The smearing on the edges is perfectly normal if you are undistorting. It's most notable when the lens distortion is the pichcushion type. Blender is deforming your image, and it fills the missing parts with the color of the last pixel on that row or column, since there is no other way to reconstruct the information of things outside of the image limits.
Here's an exaggerated example to illustrate the point:
Other programs do give you the option of making the edges transparent or black. Blender does not (As far as I know).
To solve the issue you do have a couple of options:
1- (The lazy and lossy option): Resize the undistorted image in the compositor so that the smearing at the edges doesn't show.
The downside is that you'll loose some information and the image might degrade with the re-scaling.
2- Work with undistorted proxies as a reference to create your 3D scene, but for the final render bring back the original (distorted) video
and distort the 3D scene to match the distortion of your lens:
You loose no information and your 3D scene will follow the distortion of your original video.