I can select the objects in all the modes, but I just can't deselect. (I even tried deselect in box select,circle select and the default ShiftRMB but nothing works.)
So if I select something which I didn't want to, I have to deselect the entire scene with A, and then go on again.
I forgot to mention one key thing, that I can deselect objects, but only when there are objects behind it I can't. Unfortunately my scene is a bit complex and has many objects behind each other and hence I just couldn't deselect any of the selected.