//EDIT #2: Forget this answer. This helps people using the Bevel modifier so I'll leave it here, but I should have read the question more carefully... you are beveling in Edit Mode, it seems.
Beveling dimensions are in relation to the original underlying mesh size. If you scale an object in Object Mode instead of Edit Mode, it is scaling the object to make the mesh appear in a different size, to say it simplified - the mesh stays the same size, it only appears smaller.
So you either have to scale the mesh in Edit Mode to change its original size, or in Object Mode you have to apply the scale after changing it with Ctrl+A > Apply > Scale to make the scaled dimension the original mesh dimension.
//EDIT: Just to make something clear which came up in the comments. Applying the scale makes the original mesh size larger (or smaller) and hence the bevel amount might now give different results as it is not automatically changed according to the applied scale.
Therefore it is always best to first apply a scale and then set the bevel amount to the desired size. And although applying the scale at first might seem to "destroy" the bevel, it is important to get evenly sized bevel amount in all directions, because unevenly scaled individual dimensions might make the bevel look distorted in certain directions.
The other way would be to first apply the Bevel modifier and then apply the scale. However this makes the change in mesh permanent and you can no longer tweak the bevel settings.