Seams are a very real part of most clothing items.
The main thing wrong with your UV's is their alignment.
Basically you need to add seams where they would be on a similar real world piece of clothing. I added vertical seam from under the armpits to the waist.

Unwrap and rotate the UV islands so that they are correctly aligned with the texture. Then straighten (align to X to Y) edges where ever it makes sense to do so. Pin these straightened P edges, so they don't move and Unwrap a second time.

The rest is tweaking here and there, with one eye in the 3D editor to see if it is an improvement or not.
To check for stretching sometimes it helps to switch to have a UV grid on the model. Other times (but I find a little less useful), is to enable the Stretching Overlay in the UV Editor.

To re leave some of the stretching in the original sleeves unwrap I added a seam close to the top of the arm then unwrapped a second time. Moved the two islands close together then tweaked the vertices along the seam to align the up together again then used the Weld tool (UV Editor) to weld the two edges back together again.
The result is not perfect by any means... more time and more tweaking :)

Doing this has reminded me about the UV Magic addon that ships with Blender. I haven't used it yet but there appears to be some very useful tools included. Checkout the UV Sculpt tool, illustrated in this Video at 8:20 https://www.youtube.com/watch?time_continue=533&v=BKZX3Xac7X4&feature=emb_logo )
Blend file 