Actually you have 4 different objects that compose your brick. I guess you have 2 problems: the problem that you describe, and the problem that a lot of your faces are overlaying as there are part of 2 different objects. So, in my opinion, to fix this, you could join everything in one object with a ctrlJ, then remove doubles with a W > Remove Doubles, then delete the faces you want to delete. Note that you will have to fill the hole that you leave.
It actually appears you're in edge selection mode, not face selection mode. Look closely at the bottom of the picture in the OP. Select face selection mode and try to select the face that way.