You need to increase the bitrate settings in the encoding panel. The bitrate defines the amount of space that can be used for each frame, the lower the bitrate the more the data needs to be compressed to fit within that rate.
For comparison consider a jpeg image, at 90% quality it compresses to 189k and at 10% quality it compresses to 44k. The 90% image appears much cleaner than the 10% image.
In a video the bitrate has the same effect, if the bitrate says it can only use 50K for this part of the video it needs to compress at 10% quality leaving you with a poor quality video. By increasing the bitrate you are allowing the encoder to use a higher quality compression to get a better result. Also by running at 60fps you would expect to want a bitrate almost double a video running at 30fps. You will also want the maximum rate to be higher than the bitrate.
An image at 1920x1080 using 3 bytes per pixel (RGB) takes 6MB of ram, times that by 60fps and you can easily see that the default bitrate of 6000kb/s will need to use some heavy compression.
While I am skipping the more detailed explanation, the basic concept is there, in reality many things like amount of the frame that changes and GOP size also plays a part in the final quality.
You can find many opinions on what bitrate to use, at the end of the day it comes down to your video and what quality level looks good to you and how big you want your final video file.