The border around the colour swatch indicates that it is keyframed, which is causing the colour to go back to it's animated setting. The border on the colour swatch is equivalent to the green and yellow backgrounds of keyframed numeric values.
Right click on the colour and choose "Clear Keyframes" to remove all of the animated data. The border will then disappear and your colour selection will stay.
If you only want the colour to change at a certain part of your animation, then you need to add keyframes to make the colour change when you want it to.