Blender's FreeStyle functionality is a rendering engine, which means that it is not object-based. You can only turn it off for the whole scene.
Take a look at this picture from the Blender Manual
The first box is your 3D scene with all its objects. Everything else in the process happens after the objects, their positions and materials are processed.
Looks like you are using BI. I use Freestyle with Cycles. Also I have not much experiences with using masks in general.
That being said, you could try using different render layers (but maybe not usable because of the mask) and fix in compositing. Or you could try putting the object in its own group and limit the freestyle lines to that group.