I wonder whether cloth self-collision works. I created a cloth with two materials and folded them like the first picture. I started cloth simulation, whose self-collision distance is 0.75. The cloth became strange like it in the second picture.I tried different self-collision distance, but it didn't work well.... Let me show four sets of experiments. I need cloth self-collision in my clothed avatar. Cloth self-collision doesn't work well as I think. Penetration or repel happens on pocket, collar or hip where shirt and pants (they are one mesh object) overlay.
The cloth self-collision is affected by the scale of the faces making up the cloth. This means that when you have a fairly coarse level of subdivision the self-collision Distance will result in a cloth that is quite separated whereas if you subdivide the mesh more finely the cloth is permitted to get closer before colliding. Since you only have a single face-width between your folds the self collision is forcing the sections apart in your first example, resulting in the cloth being blown out into the unnatural shape.
The solution is therefore to simply add more subdivisions. This will result in the scaling of the self-collision to be more appropriate and will also produce a more natural behaviour of the cloth - but this will also significantly affect the simulation time.
To demonstrate this working, here's my subdivided example mesh :
I first ran the simulation with lower subdivision and a Self-Collision Distance of 0.55. This resulted in overlapping faces as in your second example. Using a high Distance also caused the cloth to 'blow out' as in your first example.
However, the higher subdivisions allow the Distance to be left at the default value of 0.75 without the cloth being 'blown out' while also preventing it from self-intersecting. In addition, the cloth was allowed to collide quite close together so there is not a significant gap between the layers. This produced the following result :
This was produced with the following cloth settings :
If you still have intersection issues (particularly from fast moving collisions) you may need to increase the Cloth Quality Steps and/or the Collision and Self-Collision Quality settings (this will also slow down the simulation).