I'm trying to use the ⌥ Alt + ⇧ Shift + S function to turn a selection of mesh into a circle so that I can cut a circular hole in an object. The right side of the image is what it's supposed to look like, and the one on the left is what I get with the object I'm trying to model, ignoring the clipping, it comes out elongated. I think it's because the object is rectangular but I don't know to work around that. Does anyone know how I can solve this problem?
The problem it looks like this is that you've scaled your model in Object Mode instead of Edit Mode, which means the scale is not uniform on all axes. In this case, Alt+Shift+S stretches the circle according to the stretched scale.
To make it work, apply the scale in Object Mode by pressing Ctrl+A > Apply > Scale.
By the way: I'm not sure how you managed to do it with the Loop Tools, because (at least in my Blender) you'll get the same result no matter which method you use, as long as the scale is not applied.