1.go to node editor and enable use nodes
2.add mask node and mix node
3.in uv editor change from paint to mask
4.LMB(leftmousebutton)+ctrl add point of the curve, close curve with toggle cyclic
5.use this mask as input for mask node and plug mask node to factor of the mix node


mask could be animated as well with adding shape keys
![basic set up][1]

for rotoscoping could be use add on http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Animation/RotoBezier
http://www.blendernation.com/2010/11/22/new-feature-rotobezier/


  


  [1]: https://i.sstatic.net/UJFWc.jpg