The normals come out of the blue faces:
But if two blue faces are facing and close to each other, showing red faces to the outside, you may have the impression that normals come out from the red faces. Simply because normal indicators (blue segments) go through the mesh:
Here on left something close to what you show in your question and on right the back ...
Recalculate Outside (CtrlN or CtrlShiftN) is Face Normal orientation.
What you have marked on screen is Vertex Normal Direction.
Try search for Normal Average.
For more info read Blenders Manual bottom part.
The mesh has some inside faces, and Blender is confused by it :).
Select the inside faces through Select > All by trait > Inside Faces
Delete them with X > Faces
Credits to @John for finding the best solution for his own problem :).
You can use 2 ways, one is to select the specific faces you want to be shaded flat and then go in the Mesh menu, under Shading and select the Flat Faces.
Or you select the object and go in the Object Data Properties under Normals and activate Auto Smooth.
The latter works very well with most objects, but does not give you as much control as the first way.
Three more things to check since you already tried remove doubles/Merge by distance
is it because of smooth shading on flat object ?
go in edit mode with vertex select mode and see if there are extra vertices on any edges (You wont see extra vertices in edge or face select mode)
In Face edit mode check if there are overlapping faces which did not get merged ...