I just updated the blender version from 2.83 to 2.91 for our baking scripts, and it was running smooth in my local machine tests (GPU rendering), but when we ran the first test jobs on AWS machines (with CPU rendering), it was really slow. From 25mn to bake a few lightmaps, it went to more than 4 hours. The only thing that we changed in the code were two breaking changes from 2.91:
scene.ray_cast(layer, ray_origin, world_normal) # became scene.ray_cast(layer.depsgraph, ray_origin, world_normal) #and bpy.ops.export_scene.gltf(export_materials=True) #became bpy.ops.export_scene.gltf(export_materials='EXPORT')
In the logs, from what I can see, there's not much difference. The most suspicious difference is that in the bakes with 2.83 there was this line before each bake:
Fra:0 Mem:593.81M (0.00M, Peak 725.36M) | Time:00:15.92 | Mem:0.00M, Peak:0.00M | Scene | Loading render kernels (may take a few minutes the first time)
Which is now absent from bakes done in 2.91. Because the logs are reaaally long (90k+ lines) it's difficult to do a thorough analysis. All I can see, when searching for the lines I log before and after baking one map, is that the logged time is much slower with 2.91 and that the line stated above is absent at the beginning. There's not much more I can see...
EDIT: Comparing the documentation for BakeSettings from 2.91 to 2.83 I see there is now a "max_ray_distance" parameter, that is set to infinity by default. It might be my problem. Gonna try and tweak it.
EDIT2: the max_ray_distance parameter didn't change a thing. I was able to reproduce the issue on my computer, and observed that while baking with 2.83 uses 100% of the CPU, on 2.92 it uses almost nothing. Gonna try the 2.92 beta next and 2.90 to see how they behave
EDIT3: I tried making a simple blender project to reproduce the problem and open an issue. But strangely, the problem did not occur then. There must be something we're doing in our code that makes it behave differently than just making a simple plane mesh and hitting the bake button...