You have to work in segments, layers and sometimes different blend files linked together.
For such large models it makes no sense to have the whole object loaded at once. Split it in smaller chunks that you can work separately.
Use layers not to have all of the geometry visible at all times. Objects that are hidden save them on a different file and link them only when you need them.
If you are doing the detail of a doorknob, for example, you don't need the towers or any of the landscape.
Think of what is your final goal for this scene. Objects that are not going to be seeing in detail in the final product can be rough and don't need as many vertices as those that are meant to be features prominently.
Most importantly you need to start thinking of a strategy to render (if you are making images) or to split it in to much smaller and manageable parts for it to work on a game (if you are creating assets).
Unless you have supercomputing power at your disposal (which I doubt judging from your post) You need to fragment your scene. Close to 3 million faces with no textures yet is a lot for an average computer.
Reality has to sink in at some point and you have to realize that blender can deal vast amounts of information, but your computing power is limited, and I think you are getting close to the max.
Try to decimate the geometry, use the minimum amount of subdivision that you can get away with. Use instances instead of duplicating objects (see these links: How can I save memory in cycles when I have a lot of duplicates? and What techniques have been used to repeat geometry or other detail to the infinite horizon?)
Plan ahead so that your scene can be rendered: https://www.blenderguru.com/articles/how-to-render-a-complex-scene-without-crashing
