I want to make this sphere follow a path, but everything I try creates a giant distance from the actual path. I guess what I'm asking is how do I get the blue line extending from the path to the sphere to be basically 0 at all times?
The issue is that the curve and the object you want to make follow the curve have different origins (that orange dot you see when you select them in object mode) The calculation for the follow path constraint uses that offset to compute how the ball follows the path). That offset is represented by that blue line you mention.
To make the object be on the path, the origin of the curve and that of the object have to be on the same place:
To make your objects have a common origin do the following:
Select your path and press Shift S and select Snap Cursor to Selected, then select the other object and press Shift S and Snap Selection to Cursor.
Then you can add the follow path constraint and the ball will move to the beginning of the path.
A different way to place the ball at the starting point of the path is to enter edit mode on the curve and select tha point you want the ball to move to and snap the cursor to it and then snap the ball to that place...