1.go to node editor and enable use nodes
2.add mask node and mix node
3.in uv editor change from paint to mask(if you want mask image ,upload that image to uv editor ,mask will be placed over your image)
4.LMB(leftmousebutton)+ctrl add point of the curve, close curve with toggle cyclic
5.use this mask as input for mask node and plug mask node to factor of the mix node
mask could be animated as well with adding shape keys,if you want mask some image upload that image to uv editor ,mask will be placed over your image
for rotoscoping could be use add on http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Animation/RotoBezier http://www.blendernation.com/2010/11/22/new-feature-rotobezier/