There is no real difference. It just makes it easier to start in the units you're going to be using. This is more for making models you intend to print, or send down the pipeline to another stage of development. As long as you just remember to do it and doublecheck your measurements at the end and apply the scale to the object, it really doesn't matter.
That being said, there are several modifiers and textures that don't scale well. For example, a low resolution texture (not a procedural one) might end up looking fuzzy at a larger size. Aslo, if you have things like particle systems and complex parenting groups, it could mess the relations up, because it uses the math to calculate the relations based on the input. For example, 1000 hairs across one unit is quite a bit denser than 1000 hairs across ten units. Your model would become nine-tenths balder.
EDIT: To clarify, if you're only using hardsurface modeling on one object with simple materials, there is basically no difference at all.