I try to make clothes for my character, but for some reason, every time when I try to animate the clothes, they always start looking like this when I add cloth physics to them.
This is what I would like them to look like:
How do I fix this?
Try checking a "Self collision" box
By the way there is same question answered. Cloth - Cloth interactions in Blender Maybe this will help you.
Cloth physics is done (it appears) with something akin to a 'explicit' FE solver, which is sensitive to the speed of sound in materials. Sounds complicated, but ultimately means that the finer the mesh, the more likely the solution is to go unstable. The instability makes adjacent nodes move against each other and ripple, leading to the appearance of crushing.
Some solutions to the problem include increasing the node point mass (something like increasing density and decreasing speed of sound), decreasing the cloth stiffness, decreasing the simulation speed multiplier, increasing the quality steps, and decreasing the mesh density. Note that mesh node count isn't what is interesting, but actually the inter-node distance. The stability is governed by the smallest distance between connected nodes in the mesh. With a static image, all solutions will be of use, but if trying to include dynamics, many of these solutions will not (will change the dynamics) and only increasing quality of simulation, of contact collisions, reducing mesh density, and using things like collision clamping controls will be of use.
With the low mesh density, cloth benefits from using smoothed rendering of faces.