What you need is a Mask to get the 2 images separated cleanly. You can obtain one rather easily from your Noise Texture by either using the Fac output Connected to a ColorRamp, or convert your rust "color" back to B&W and do the same thing (pictured below - using the colors preserves the same cutoffs you had before, but may not be necessary as I had to further adjust with another ColorRamp anyways). Regardless, set your MixRGB node back to "Mix" (from "Difference") and connect the "mask" as a Mix Factor. If you want even sharper definition on the cutoff areas, consider switching the mask ColorRamp to "Constant". You can also see I had to invert the sliders on the mask ColorRamp to make the mix correct - another way to accomplish the same thing is to switch the positions of the color inputs to the MixRGB Node). Lastly, the RGB to BW node is not necessary - I just put it to highlight that the mask must be converted back to B&W:
Here's another example showing some correction (no RGB to BW, and the color inputs are connected "correctly"). If you want the most detail in your texture mix, you will have to incorporate your Mask into your Roughness and Bump maps as well (shown below).
However, this method can get complicated if you have a lot of maps to mix. An easier way is to keep each material as part of its own separate shader, and then mix the two using a Mix Shader and the same Mix Factor.