I'd suggest to do it the other way around: Having diffrently colored objects in a collection and instancing them via the particle system.
In the render tab of the particle system choose reender as collection.
Option 1: pick random. (even distribution)
Option 2 if you want more controll how often the objects are instanced in relation to each other you can use the "count" feature.
(Option 3 for even more controll use the AddOn Animation Nodes)
This intent you have is a advanced procedural workflow. I think its fun to work like that but it needs a bit of trickery often. As far as I know Blender has no native functionality like that, because this is very specific also, and blender is not a fully procedural tool.
The AddOn Animation Nodes is a gamechanger in terms of making the python interface more acessible and easier to handle.
Here is a barebone example in AN (Animation Nodes) that changes instancing collections in the particle system. The boolean value (See A) would get your logic as input that does the color allocation or gets the color from the material and sets the corresponting partile collection.
Going this far with animation nodes, maby it is simple to use it also for color allocation. At the moment you do it with ObjectInfo.random wich gives you no controll. If you have multiple Color-Sets you could smoothen the workflow with Custom Object Propertys that AN interprets an does the rest.
Then the only thing you would have to do would be to duplicate the object and change its custom property.