Object transforms are sort of like a non-destructive "offset", which is applied on top of the raw mesh data.
When you scale (or translate, rotate, etc.) an object in object mode, the actual mesh data is not affected. You can press AltS to clear the object scale at any time and go back to the basic mesh (similarly, AltG and AltR clear location and rotation).
So when you edit an object in edit mode, the object will appear with the object transforms, but you are editing the mesh data directly (without any transforms).
To visualize this, I'll use extruding, since it's effect is a bit more obvious than beveling.
What happens in the gif:
I extrude the faces of the normal cube. As you can see, the extrusions are all the same size.
I then scale the originally unscaled cube along the X axis. The extrusions along that axis are larger, since the mesh data is being "stretched" along the X by the object scale.
I extrude the faces of the already scaled cube. Naturally, the two cubes are now roughly the same. It's sort of like I'm editing the normal unscaled cube which is then scaled up on the X each time it's displayed.
I undo the extrusion on the scaled cube and apply the scale.
By applying the object transforms to the mesh data (CtrlA), the underlying mesh data is modified according the object transforms (which are then reset).
Now extruding the no-longer-scaled cube results in even extrusions.
Note that object scale only has this effect when it's not uniform across the axes. For example, a scale of
3, 3, 3 will not stretch the cube and make bevels or extrusions skewed, as all the axes are being stretched the same amount.
This goes for modifiers too. Modifiers operate on the base mesh data before it's adjusted by the object transforms.
Ctrl+Ain object mode) $\endgroup$