I approached it as an integer sequence: Took your short random N-digits long sequence and put it several times in a row (see the table row A), then I took the source of random values again (with different seed) but this time I put each value N-times in a row (B), then I summed it up (Sum) and finally calculated its reminder after division by N. This way I kept the mutual uniqueness but mixed them at the cluster level (Mod).
The last row when divided to rows by cluster:
To implement it in GN I used your randomizing object two times (but I actually duplicated it and made seed as a modifier input attribute to improve randomness) and utilized the fact that depending on whether the object is created "As instance" or not, orders points in different manner (actually like A and B in the table). Then it was just a simple arithmetic as described above.
Randomizing only one row and then copy it is easy and can be made in many ways. I implemented a switch controlled by a modifier attribute. See results for switch on and off bellow:
It is not perfect. I am not so good at modular arithmetic to tell how good pseudo-random numbers it provide, but the fact is there are some seeds where it looks suspicious. Actually even your GN randomizing object isn't perfect - it works for given seed (0) just because of pure luck. Actually there is non-zero probability it returns just one vertex. To reduce this probability I recommend to increase the number of generated points to some humongous number.