Is there any ways to speed up cloth simulation? I have fiddled with multi-threading count but maybe I am missing an obvious option. Is GPU cloth simulation a thing?
You can transfer lower-poly simulation into high-poly mesh with Surface Deform modifier. Lower-poly or mid-poly sim is also easier to control and weight-paint.
if you don't already have a lot of detail on the shirt you could try using a low poly and choose a frame you like, then raise the poly count and add the details afterwards.you can also try lowering the simulation speed and quality which can help lower the time taken to simulate without making it unstable like just lowering the quality might.
for the shirt rolling up the arms when you simulate, try lowering the outer face thickness of the character, if you do this the cloth will collide closer to the mesh but is more likely to clip through so raising the the simulation quality and lowering sim speed will probably be necessary to get the result you want.