If you turn diffuse bounces to 0 (disabling indirect light), the source of the mystery reveals itself.
Here's 100 samples with no portal, and no diffuse bounces:
And the same, but with the portal:
So why is the full scene noisier when we turn on diffuse indirect light? The portal INCREASES the amount of light on each sample, thus increasing the number of diffuse bounces that return something instead of nothing. So you get more noise there. The combination of the small window and white walls mean the scene is almost entirely indirectly lit.
There's a few things you can do to speed this up. First of all, the portal is much bigger than the window. Shrink it to exactly the size of the window. And while we're at it, throw a solidify modifier on your room cube, that way you don't get the weird light leaks at corners. That's imperfect though, so you'll need to fake a lot more than that. Time to stop world indirect light entirely. Go to the world panel, and change bounces to 0:
The portal will still throw sunlight on the wall, but it won't keep bounceing around and making a ton of noise.
Finally, replace the missing lighting by making a copy of the portal (leave it in the exact same place), disable the portal option so it's a regular area light, and give it a sky color. Adjust power to taste. That gets the indirect lighting clean enough that the denoiser can take care of the rest. You can disable desnoising on all passes except diffuse indirect if you like:
And the result of all that, same 100 samples:
You don't have to use ALL these techniques of course, you use just some to get more realism at the cost of render time. For example, you could set world bounces to 1 instead of 0. Play around to get the performance/quality balance that works for your shot.