There is a plane that seems to appear just on this face. In edit mode this "flat plane" doesn't show up, however when I go into object mode it appears. I have added a boolean modifier for the top part (as seen in the photo). There are no creases or bevel weights, and I have "merged by distance" if this helps.
I initially assumed it was a normals issue, likely caused by a hidden face or pinched edges, but we can see the outline of the face sticking out of the profile while in object mode, so I'd assume its not a hidden face or normals issue.
But just to fully rule that out, go into edit mode, edge select mode, and deselect everything, then go to Select > Select all by trait > Non Manifold. and ...> Interior faces, if anything selects then you have some interior geo.
So next assumption, I'd assume it's caused by the boolean modifier, if you turn off the boolean modifier does the extra face go away?
If so could be caused by a calculation glitch with something on the boolean object, so common things to try are to simplify your boolean object (ie apply on any modifiers so its less things to calculate at once), or could be from the boolean objects normals being to closely aligned to your main objects normals, so make sure your boolean object is a little larger then the hole its making needs to be (ie give it some overhang on the sides and top, where the hole its creating needs to be).
Also you mentioned you did a "merge by distance", I assume that was on your main object, I'd suggest doing that to your boolean object as well, so try and rule out it being an issue with the boolean geo.
Hopefully this is of some help.