This is a classic weight issue. The vertices that aren't moving correctly are either weighted to the wrong bone, aren't weighted to a bone at all, or aren't weighted enough to the right bone.
Vertices follow the movement of the bone they're assigned to, but they can be assigned to only follow a percentage of a given bone's movement. For example, if you assign a vertex to a bone at 50% (or 0.5) and then rotate the bone 90 degrees, the vertex will only follow half of that movement... 45 degrees. Furthermore, you can assign a vertex to more than one bone. For example, you might assign a vertex on the knee to 50% on the lower leg and 50% on the upper leg so that smooths between both of them.
It looks to me like you have a handful of vertices on the (left?) leg that are not assigned to the bone going back to the left. Consequently, the bone moves to the left, but the vertices don't follow. A similar thing is probably happening with the ones on the right. Note: "not assigned to" might also mean "assigned to a different bone."
Interesting note: it's possible (depending on your Blender settings) to have a vertex with more than 100% total influence. Unless you have "normalize all" turned on, all the weights will get added together and do weird things.