There are a few things going on, here.
Firstly, the z-depth is exactly the distance of each pixel from the camera. You can demonstrate this by setting the clipping planes of your camera to specific values, and then using your mouse to inspect the RGB values in the render result view (not the saved image). The closest pixels will be the same value as your near clipping plane (assuming there's geometry that close), and the farthest pixels will be the same values as your far clipping plane (except for where the sky is visible, where the value will go to "a very large number"). I proved this to myself just a couple of days ago. I'll see if I can dig that project back up and upload a demonstration of some kind.
Secondly, those values don't fit in any typical RGB image. So anytime they need to be put into a file, they get mapped to 0-255. That means what might have been a 19-1200 floating-point range got squashed down to 0-255 non-floating point.
Lastly, when the image undergoes that transformation from floating point precision to 0-255, it ALSO gets mapped to a new color domain, which often includes a gamma adjustment and other strange transformations. It's usually quite effective for visual data, but plays havoc with numeric data.
The solution is an image format that is not limited to a 0-255 range for each RGB value. Enter OpenEXR and HDR. They can store the raw, floating-point values as they are in Blender, without compressing them... much. They're still limited to 32-bit floating point values, but that's plenty for just about anything.
So, if your data ever leaves Blender, it's been compressed somehow, unless you save it DIRECTLY to EXR or some other high-range format (without any intermediate nodes or processing). The best idea would be to use the buffer directly in Blender, rather than loading it from a file. If you do need to save it to a file, make sure that you're saving to some 32-bit format.
It may also be worth looking into the Mist Pass. It's similar... but different.