Option 1: Setting with Scroll wheel
If you use the scroll wheel while in bevel mode the bevel uses more edges, giving the corner a more rounded look.
Option 2: Using the advanced tool menu popup:
There is also another way (which doesn't need a scroll wheel) to set more advanced settings for the bevel function (actually works for any tool and add primitive actions).
After you did the tool action (in your case bevel) a small popup is seen in the left corner of the 3d-view screen (in Blender 2.80) which you can click to expand it. There you can modify the action you just did.
PS: In Blender 2.79b this advanced tool menu was located in the tools tab
Option 3: Use the +/- keys
As quoted from the comment from @lemon you can also
"use + or - from the keypad." - lemon
to increase the segments of the bevel tool.
Happy Blending! :)