When applying triangulation modifier, what kind of algorithm does Blender use when choosing the "Beauty" option for quads or polygons? Is it Delaunay triangulation on individual polygons?
After a quick look at the source code, I believe it tries to find a triangulation that leads to the best angles for each triangle (ie biggest minimum angle and smallest maximum angle).
For a quad, there are only two possible splits so it tries both.
For a polygon with more sides, it does some sort of greedy iterative search of an initial triangulation and edge rotations.
Possibilities where triangles that are flipped relative to the initial polygon are discarded.
- bmo_triangulate_exec in https://github.com/blender/blender/blob/master/source/blender/bmesh/operators/bmo_triangulate.c
- BM_mesh_triangulate in https://github.com/blender/blender/blob/master/source/blender/bmesh/tools/bmesh_triangulate.c
- BM_face_triangulate in https://github.com/blender/blender/blob/master/source/blender/bmesh/intern/bmesh_polygon.c
- BM_verts_calc_rotate_beauty in https://github.com/blender/blender/blob/master/source/blender/bmesh/tools/bmesh_beautify.c
- BLI_polyfill_beautify in https://github.com/blender/blender/blob/master/source/blender/blenlib/intern/polyfill_2d_beautify.c