1.go to node editor and enable use nodes
2.add mask node and mix node
3.in uv editor change from paint to mask
4.LMB(leftmousebutton)+ctrl add point of the curve, close curve with toggle cyclic
5.use this mask as input in mask node


mask could be animated as well with adding shape keys
![basic set up][1]

for rotoscoping could be use add on http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Animation/RotoBezier
http://www.blendernation.com/2010/11/22/new-feature-rotobezier/


  


  [1]: https://i.sstatic.net/UJFWc.jpg