As well as keyframing masks you can get a lot of control by making your own animated video strip and using it as a mask strip. CGCookie has a archived tutorial that covers making and using the mask video. The only other tutorial I know of is by David McSween, he uses a single greyscale image and animates the brightness and contrast to get the transition.
The basics of creating an animated video to use as a mask is to create a video that defines the transition, black shows the old strip, white shows the new strip with levels of grey a mixture of the two. You can easily imagine using blender to create a simple animation of shapes for this, add this transition video as a strip in the VSE and then add a mask modifier and select the transition strip.