I am guessing, but here's one problem. Have you parented the objects properly so there are object origins or, ideally, empties where the rotations take place? Because I guess you haven't just rotated stuff, you have also moved it even though you didn't mean to.
Blender doesn't remember around what you have rotated objects. Sometimes it rotates around the cursor, sometimes around a common selection center. For key frames, that is irrelevant. Key frames only care about the object origin mostly. When you rotate more than one object, it's almost always also movement.
If you don't use armatures, put empties where your box is supposed to fold and create a hierarchy of parents and children. That way you can rotate the inner part around the empty and all other parts will follow, even if they're not animated.
An example:
The sides of this unfolded box are 6 separate objects, similar to your setup. I just used a lot of empties as well.
Boss is the main empty. If I move that, all others follow.
e1 - e4 are all parented to boss and e5 is parented to e4. If e4 is rotated, e5 behaves as if glued to it.
That's also the reason why my top flap does a 180° turn. I made e4 turn 90° and e5 turn 90° as well and it adds up.
The empties may be invisible, but they're handy pivot points and all the faces are parented to the empties, so they move with them. That's the result when animated:
You could achieve the same with bones. Then you could model your box out of a single mesh and deform it by posing the bones. But empties are fine, too.
