TL;DR:
To put it simply, after n light bounces, Cycles switches the material shader to a simplified diffuse material (i.e. AO), instead of evaluating the full material node tree.
This Simplify to AO feature can be replicated with this node setup:
Note that because the math node is set to 'greater than 1', the AO node will only kick in at ray depth 2.
Why?
This often give a significant speed up because we author the materials to look good to the camera, all those normal maps, fresnel effects and fancy edge shaders don't really change how the light behaves after a few bounces. For bounced light, all that matters is how reflective the material is(i.e. its albedo), does it bounce back 10% of the light, or 99% of the light?
Here is a more elaborate example of the optimization at work.


We can easily cut the render time in half by falling back to a diffuse BSDF after 1 bounce instead of whatever megashader we are using. This test is an outdoor scene with bounced limited to 3. The rendertime improvement will only get better if there is more bounced light, as with most interior scene.
AO distance setting definitely plays a role in the output image. But it is unclear how other ambient occlusion settings (factor and world color) affects the AO pass.
Effect on perfectly specular surfaces
A question was raised about how this will affect perfectly glossy surfaces. To answer that we can assemble a test scene with 2 parallel mirrors with a subject in the middle:
This is with the bounce turned all the way up to 8. We can see precisely 8 monkeys before the mirror turns black.

If we turn on Simplify to AO to 2, it seems the shader returns the environment color after 2 bounces instead of returning black, which is arguable more energy conserving and correct.

A final comparison: this is with the global bounce setting turned down to 2. Notice all reflection rays after 2 bounces are black.

This is an extreme example, with normal world scenes, it's unlikely we'll need anything with more than 2 bounces.
Take Away:
Always set AO Simplify to 2, or just limit your global bounces to 2 or 3.
If you have a complex shader network, create a diffuse shader for all the non-camera rays. This can be faster and more accurate than the AO simplify option.