Sorry for the necropost, but hopefully this is of use to someone...
I'm using WebGL here, but what I've learned may help with blender/game projects... I used this geometry suggested by Duarte - mainly because a system of bend, shear and structure constraints intended for a square cloth were not working (the hypotenuse of the sail triangle was too loose compared to the rest) … and the Duartes geometry suggested above works much better.
I attached the lower part of the sail to a boom with follow-pins, hinged it to the mast using a universal joint and then used a verlet rope to control the booms lateral movement (the mainsheet) - and an overstretched spring to hold the boom down (a boom vang)
It works well. The rope goes loose when the boom crosses the deck, and then settles on the other side of the boat at the same angle. I can, of course, pull the boom in or let it out by adjusting the constraints for the rope segments. The vang (pulls the boom down) gives a very nice motion and really good sail shape.
To reef the sail, I use several sets of precalculated pins that hold the reefed sail folded on the boom. I tween between these whilst raising and lowering the mast pins and it gives a very smooth and effective 'reefing/unreefing' animation - the wind still moves the loops of sail that are reefed, but not much due to all the pinning. Just enough to prevent it looking static. The pins holding the reefed sail down are tighter at regularly spaced points down the boom - this makes a nice effect that looks a lot like sail ties.
Regular wind forces don't work well.
Instead I calculate wind from the mast, backwards. The mastward node receives the global wind vector … but as I work backwards each node I come to I adjust the 'local' wind vector to bring it very slightly more in-line with the sail (as though the sail redirects the wind flow) … this gives a proper sail shape, with the majority of the curvature in the leading third, and flattening towards the trailing edge... more importantly, the changes in wind direction across the sail allow me to trivially calculate the reciprocal thrust vectors which can then be averaged and applied to the mast (it's a cheat, but it is surprisingly realistic and saves trying to calculate lift using a costly laminar flow model)
(if you do this, you can model the keel too, to progressively cancel out the roll component and permit only the forward component to act on the vessel. Alternatively, you can just reduce the lateral component and apply the resulting vector to the boats acceleration through water)
I've now moved on to the jib, but found that it can be tricky to control the stretch well without imparting unintended energy. As it is a far less constrained sail it's a little more tricky to get its cloth simulation balanced nicely - but I'm slowly getting there. Again, verlet ropes.
I've found that ropes prefer multiple overlapping constraints to avoid them acting like string. One between neighbours, one between neighbours-but-one, and one between neighbours-but-two, etc.... to about 8 orders. This helps the rope make nice curves with no sharp turns - which means they 'whip' properly when the sail changes sides and also fall nicely and naturally on the deck.
Penants on the stays and the flag on the stern are trivial, but really help to sell the effect and help show the wind direction more intuitively. I only move the pins with the boat - the unpinned nodes are left to catch up.
Finally, I find that the sail SHOULDN'T be parented to the boat either. Only the pins move with the boat - the free sailcloth nodes should be moved forwards with the forward motion of the boat but NOT adjusted with the roll of the boat - this allows the sail to fill and unfill slightly with the roll of the boat … again, I find this really sells the illusion. I tried allowing the sailcloth nodes to catch up with the pins naturally like the flags and penants do, but found that this affected the sail shape horribly, so now I move the nodes forward with the boat to too much avoid stretch, whilst not correcting them for roll to allow some 'slop' and 'slap' in the sail.
I hope some of that helps someone looking to script their own verlet sailcloth simulations in blender or in game code.
I just wish I could get the sound right. I haven't a clue about making dynamic slapping sounds when the sail ruffles or pulls taut... but I figure I can find an "audio guy" if I get something near production quality.
Application is a sailing simulator with a focus on RYA sailing, navigation and docking topics, as I'm currently taking my Yachtmaster Ocean qualification - and couldn't find any accurate simulations out there that deal with all aspects of the syllabus.
Anyway, I hope some of that gives readers some ideas or inspiration. Good luck with your models, whatever level of accuracy you're aiming for : )