So there are a few problems that I'll mention, but the real problem is just that you're expecting too much from Blender rigid body physics.
First thing we want to do is check for unapplied scale on any objects. In your case, yes, there is at least one object with unapplied scale. This will screw up rigid body simulations. So we'll apply scale on all objects.
Second thing we need to do here is look for any reversed normals. Normals are how your physics simulations know what is the inside and the outside of a mesh for purposes of computing collisions-- that is, what is a rock, and what is a cave? If we turn on face orientation in viewport overlays, we can see you have a mesh with reversed normals:
We can fix those normals by entering edit mode on the offending mesh, selecting all, and recalculating normals (ctrl n for me.)
Third thing we'll do is set the rigid body physics cache to something larger than 3, so we can actually get enough frames on the simulation to see what's happening.
But the real reason why it's exploding is because meshes are starting in collision with each other. If we look over here:
We can see that those meshes are already in collision. Where are they supposed to go? They get propelled through each other because it's the only way to get them out of collision.
Now, in this particular case, we have an RB constraint with disable collision, so this particular pair is okay-- but your plan is to have them break apart later, and the RB constraint is fixed, so if they're in collision now, they'll still be in collision later. Any time we do anything to enable collision, this structure is going to explode.
And, not every combination of objects has a fixed joint disabling collisions between them. The collisions that are not disabled by constraints are what is causing the explosion at frame 1 (at least, after taking care of scale and normals.)
However, even if you get these meshes out of visual collision, they are still going to be in collision, because "mesh" style collision-- which is your only option here, considering the concavity of the meshes-- isn't based around only the mesh that you see. Instead, mesh collision has an invisible thickness, which it needs, because otherwise, everything would just teleport through everything else.
Indeed, at this point, if we adjust the objects to take everything very clearly out of collision with each other, our physics work properly:
Our objects fall straight down, just like they're supposed to, rather than exploding on frame 1. The only problem is, we don't want to adjust their positions like this.
The reality here is that your meshes are too detailed, and too close, to use your unedited mesh objects as rigid body colliders. Instead, use physics on non-rendering abstractions of your rendering meshes, and then parent your rendering meshes to those objects:
If we use physics on these cubes, rather than on our rendering meshes, we can use "box" collision instead of "mesh" collision. Box collision is much more reliable and performant than mesh collision. And the collision is what-you-see-is-what-you-get: there's no hidden margin that surrounds the collision. The cost is that the collision doesn't perfectly represent our mesh, but we can approximate that mesh to whatever level we need for our particular render.
Note that boxes are not necessarily the best choice-- what's important about boxes are that they're collision primitives. That is to say, they're calculated mathematically. Other collision primitives are sphere and capsule, and either one of those will work as well, and give you different behavior. You should pick which primitive to use on the basis of whichever best represents the shape you're trying to emulate. (Cylinder should be a collision primitive, but it's a non-standard one, and I've had some bugs with it, so I'd recommend sticking with capsules instead.)
Now, as for using a big scale for physics, there are two reasons to do this. One is that it shrinks that invisible margin used for mesh collision. That's a mixed bag-- you get collision that more accurately matches your visual mesh, but at the cost of rigid bodies teleporting through each other on collision (at a velocity, relative to their size, that they wouldn't if smaller.)
The main reason that using big objects leads to better physics is because the distances are larger, so the forces are smaller, relative to the size of the objects-- like gravity. At a 1m=1unit scale, Blender's gravity is Earth gravity. But if we use a 1cm=1unit scale, Blender's default gravity is only 1% of Earth gravity. Smaller gravity means that meshes are less likely to acquire high velocity, so they won't teleport through each or penetrate too deeply before Blender notices the collision; they won't stretch on their constraints to the point of breaking before Blender notices and pulls them back where they belong.
