Use a property switch
Declare a boolean property with an update method.
def toggle_op(self, context):
if self.toggle:
bpy.ops.foo.bar() # start the operator
bpy.types.Scene.toggle = BoolProperty(update=toggle_op)
In your modal
if not scene.toggle:
return {'CANCELLED'} # or self.cancel(context) to do some clean up
Related example for 2.7x