Sorry if I misunderstand your desired setup (including your use of the math node), but try something like this and let me know if it works for you - It "works" the way I think it should, but I'm not sure that matches what you have in mind. If it's no good, I'll delete:
