How to fix Cloth Simulation from this odd compression

Question

The objective of this simulation was to just have the cloth hang down so part of the tent door is open and part is hidden behind the tarp. Originally it was a simple plane, I have recently solidified it (0.01m) after reading some tips and answers from other questions on StackExchange but this did not solve it.

I will provide all the images showing my modifier stack, cloth physics settings, collision settings and the static pin group as well as the blend file below.

I have had to delete the bakes for the simulation due to file size for blend exchange.

Answer 1

If the fabric curls up like this or explodes the collision detection is usually to blame. But it takes only a few steps to fix this.

1. Add a floor: Add a plane, scale it up, add & apply a Subdivision modifier (3 levels, type: Simple), and add a Collision physics to it
2. Fix the door:
   1. Swap the order of the modifiers to 1. Subdivison, 2. Cloth ←→ 3. Solidify
   2. Change the levels of the Subdivision modifier: 2 (Viewport) and 2 (Render)
   3. In the Physics tab, use the Silk preset. For some reason, your door plane is way too stiff. (Presets are in the drop-down menu right next to the Cloth section; the icon with 3 dots and lines).
   4. Change the Distance of Object Collision and Self Collision from 15 cm to 5 mm (0.005 m). The default value of 15 cm is too high for your tent in real-world size.
3. Optimization: Remove the Collision from the 2 branches objects. It has no effect on the result since the fabric slides only briefly over the left branch and is attached to the right branch anyway. But the removal will speed up the simulation.
4. Run the simulation. It should run smoothly. If you get single vertices that go crazy for a second ("spikes"/"fireworks") increase the Cloth → Quality Steps from 5 to 9 and they should calm down.