The Orthographic Scale factor represent the maximum dimension (in scene units) of the portion of space captured from the camera.
So a scale of 1, means that a square plane whose size is 1, would perfectly fit the camera's view in the mayor dimension (for the default 1920x1080, the wideness).
If you double the scale, you'll get half dimension (1:2 scale) because the way it's mean to work like the traditional scale map. Maybe it's even easier to think in these therms:
The scale of a map is the ratio of a distance on the map to the corresponding distance on the ground.
About your specific problem, I think I would suggest the following workflow:
- Match the camera's and reference image proportions and import as background image
- Align the view
- Tweak the scale factor till you are satisfied