You don't necessarily have to convert video to an image sequence.
While it is possible to decode and track without having to use an image sequence, some videos will work and some will result in errors, depending on how they are encoded.
Blender is not broken, but can only work with video that is encoded with certain specific parameters.
Video encoded with intraframe compression will work fine. Each and every frame containd all of the information, so it does not depend from the
information of previous frames or any of the frames that come after it (same as with an image sequence).
Video encoded with interframe , or long GOP compression, will likely yield errors, since the information needed to decode each frame is dependent on adjacent frames, and only the differences are calculated. In other words, there is a key frame that has all the information, for the next few frames only the differences are calculated, but to decode those frames you need to reference the keframe everytime.
Also, if the video is encoded using any kind of variable framerate (some phones use this to make small files, as well as some screen recording software), then it needs to be re-encoded and conformed to a uniform and stable framerate, before being imported in blender.
Blender will not do on-the-fly framerate conversion or temporal interpolation in any way.
For more information read: http://www.leckman.com/articles/codecs_05.html