Theres a few aspects to what I am looking for in terms of solution.
We deal with extremely large scenes, and take huge advantage of instancing, dupligroups and particle systems
However, when you have thousands of objects (not particle systems) in a scene... blender starts to go slow.
Our initial idea was to convert the instances / dupligroups over to particle systems... that way its one object which has many instances, instead of many objects.
We have looked into trying to convert dupligroups / instances into a particle system, however... we cannot seem to alter it via the python api (https://developer.blender.org/T48022). Is there a efficient way of storing thousands of objects with locations / rotations without actually having the slowdown that normal objects would have? Is there a way of being able to create non-object instances via python?