When I'm the camera view mode, the environment texture shows up correctly, see:
However, when I'm controlling the viewport freely, the environment texture doesn't work at all but gets replaced by a random color:
I remember it used to not work this way, but I don't remember what I'm doing differently from before. The background settings I'm using are visible on the right of the picture. I'm also using Filmic blender.
Thanks in advance.