1 - Import your photo as "Image as planes", create an object for masking purposes and assign him the "Wire" maximun display; set a rendering size (maybe the same as the photo), create a camera, set it to "ortographic view", clear her position and rotation (ALT G, ALT R), move it on the Z axis to have it over the photo (distance doesn't matter in ortographic view, control the size of your camera view with the "ortographic scale" instead).
2 - Press numpad 0 to enter camera view, adjust the mask, assign the mask to layer 2, create a new render layer, assign layer 1 to render layer 1 and L2 to RL2; assign an emission material to both the object (cycles) or shadeless (BI), intensity 1, color of the mask pure white, turn off any kind of lighting and render the scene.
3 - Switch to the compositor, duplicate the render layer node and set his output to Render Layer 2 (the mask). Use this output to control a "set alpha" node. Normalize is for assuring that black is pure black and white pure white; Invert to invert the effect of the mask; Fast Gaussian blur to have some "Feather edges" effect.
Works also with movies and animated masks!