I created 2 masks in Blender's mask editor. In 1 frame of my video I created the initial mask points by using Ctrl-Click. In the other frames, I dragged them to the correct position. In the node editor, I used the "Feather" option for both of them (but I don't see any difference with or without it).
Afterwards, I noticed that one mask is anti-aliased, while the other isn't:
How can I make both masks anti-aliased?