It seems that to do this you need to be looking from the Camera View and adjust the Camera settings.
Set your Camera to Orthographic Mode
The clipping Start and End will be controlled from these Camera settings, not the Viewport clipping settings.
Align Camera to View
CtrlAltNum 0
Rotating the view
Enable "Lock Camera to View"
(It's in your Properties Panel)
This lets you look around without popping out of Camera View.
Zooming the view
To zoom in and out you will probably need to move the Camera along its local Z axis by grabbing it. [If anyone knows a better way, please chime in in the comments] To do this, select the Camera, Grab G, then press ZZ and adjust using your mouse. Left Click to accept the position.
Note: Your Camera must be on a selected Layer in order for it to be grabbed.
Demonstration:
Some thoughts on workflow
If you will be spending a lot of time working like this you might want to keep an Outliner window up that always shows your "Camera" object so you can select it again any time you need to. You can do this by simply typing "Camera" into the search field and leaving it that way. You can still use the Outliner for other tasks in a separate window space.
If you find selecting and grabbing the Camera cumbersome, it may be worth considering whether you can see what you need to see by using another approach, such as hiding objects or masking part of objects using the Mask Modifier.