Setup a few extra luxuries on your camera. Such a Depth of Field, which can sometimes be a deal breaker in photorealism. Set it up and see instant realistic results. Play around with the focal length of the digital camera on blender, see which one suits your scene better. There are tons of little settings that you can adjust to make it more believable, but ...
When separating colors, it is separating the primary colors.
This means red's hex will be: FF0000
Blue's hex will be: 0000FF
Green's hex will be: 00FF00
This means if you have a hex of 00FBFF and separate the colors, and only keep green, the 00FBFF will become green, because it's still got the G in it.
So when trying to change the color blue, you can't ...
For anyone that stumbles upon this page, I had posted the question in another forum and wound up working towards a solution there. I will put a link below so you can get to it. Thanks to anyone who gave it a shot, I appreciate the help!
Blender Artists Answer
Hard to tell without seeing your setup, but my guess is : the mask plugged in Core Matte always output a 100% value, while the rest of the shirt output a lower value ; ie, there is some green in the shirt and the keying node make it a little bit transparent. Nothing noticeable at first, but noticeable by contrast with the mask.
You can verify that by ...