How to find non-manifold geometry
From within Edit Mode, using Vertex selection, with no geometry selected, use the menu to navigate to Select > Select All by Trait > Non-Manifold. (The keyboard shortcut is ShiftCtrlAltM)
This will show you which part of your mesh is non-manifold.
What is non-manifold geometry?
Geometry is non-manifold if it doesn't form a single volume. Examples include internal faces, edges that don't form a face, vertices that don't form a face, two volumes connected at an edge or at a vertex.
Why booleans are not (usually) the answer
The problem is that you tried to model everything with booleans instead of actually taking the time to model a proper mesh. To make things worse, some of your geometry was very dense, and very dense geometry is more likely to cause problems when booleaned.
Instead, try to model it one area at a time, using as many quads as possible:
I'm oversimplifying with labels of "bad" and "good" for the sake of simplicity here.
The Mirror Modifier is just there to save time modeling, since this is a symmetrical shape.
Here's a list of the operations I used to create the mesh in my example: Extrude, Loop Cut, Delete (faces), Fill (faces), Snap (to vertex), Merge, Add (Circle), Bridge Edge Loops, Remove Doubles, Recalculate Normals, Crease. While that might seem like a lot to learn, these are operations you will use over and over, so they are worth getting comfortable with. It's likely you know some of them already.
While in Edit Mode you can select an edge and press ShiftE to add a crease to it. Typing 1 at this point gives the edge full sharpness and keeps the edge sharp even when a SubSurf Modifier is on the mesh.
You can later readjust the crease weight under the Edge Data section of the Properties Panel (a.k.a "the N Panel"):
Tip: In Edit Mode you can select an Edge and press ShiftG to select "Similar" edges, like those that have the same "Face Angles", and in this way select all the 90° edges at once so that you can crease them. It saves time.
There is no real substitute for a mesh with good topology. By learning to create better topology you will be able to make many more types of models, without encountering these kinds of problems. There may be some cases where you can get away with using booleans, but keep your mesh density lower in order to reduce the risk of problems like the one you experienced.