I have found the solution . It turns out all that needed is to focus the view to the selected object.
How to focus on the selected object?
First , go to object mode and select the object , then view > align view > view selected . The camera navigation speed will return to normal.
To deal with small part of a large object where the view move too fast , go to edit mode and select at least one faces of the small part , then focus the view , now the camera speed no longer move too fast.
The same method is applicable if the camera move too slow (eg : switching from small object to large object) , simply refocus to the new object or mesh and camera speed will return to normal.
Here are the gif showing the problem and the solution. My blender version is 2.79 , blender UI might be different for blender 2.8x and newer , but the solution is still the same .
It is a very long gif , watch until the end.