Glossy shaders won't affect the z-pass at all. The z-pass is literally a gray-scale of how far away each pixel is.
The answer to this question is exactly the same as any other question dealing with Cycles noise (though it might not seem like it at first). Noise in regular renders is a precision issue, and the noise here is also a precision issue. The way that Cycles improves precision is with more samples.
It's slightly different with the z-pass in that there's not nearly as many contributing factors to "how far away is this pixel" as there are to "what color is this pixel." But there are some. The main contributor is likely anti-aliasing. But even then, the solution should be the same: render more samples.
cegaton's recommendation to adjust the near and far clipping planes, is also a good idea.
If that doesn't help... well, I may just be mistaken... and that's okay.