I want to be clear, I am not an expert in 3-chain IK. I tend to do character stuff, with what are essentially two-chains. Part of the reason I encouraged you to ask this question is because other people on this site might have better answers than me-- both of us might benefit from hearing what they have to say!
So, sometimes, continuous rotation is mathematically impossible. Not just hard, but impossible.
The two bone chain on the top cannot ever change the bend of the "knee", because it needs to rotate through the straight, but it cannot reach the IK target when it's straight. It will jump between the left and right rotations to reach the IK target because the straight is impossible. Same goes for the bottom one: to change the bend, it has to rotate through an impossible straight. If we want to smoothly change the bend in an impossible situation, we have to cheat: move the IK target, drop to FK control for a moment, move or scale the bones, or rotate through an axis that we don't generally want our IK to bend (a human knee isn't a perfect hinge, it's got a bit of tolerance.)
Still, there are times when a continuous solution is possible, but Blender will never give it to us, no matter how many iterations we tell it to use:
We're changing the direction of the bend of your third bone. On the right, we can see what Blender's IK gives us: totally discontinuous. On the left, we can see that there is a continuous solution to this problem.
We could eventually tune everything on the IK system to give us this continuous solution, carefully rotating bones, carefully timing where we keyframe-- but man, what a pain!
So what am I doing on the left to generate a continuous solution here? I've just added a few extra, non-deforming bones into the IK chain:
I've sandwiched in two extra parents, the "mch" bones to the original "def" bones. Check the outliner for hierarchy if you don't know what I mean by sandwiching. These bones have no rotation relative to their parents-- I've created them by duplicating their parents and snapping to tails-- so they can rotate either direction; they retain the axis locks of the rest of the structure, so their children maintain the same proper rotation we want. They act to kind of give their children bones negative IK stiffness, and rotation that is bend-agnostic: some of the IK rotation can be applied to these bones instead of the deforming bones, rotating smoothly through transitions that Blender can't handle otherwise.
You can see that I've only animated these mch bones. I animated two, but the second was just to match, as closely as reasonable, the final orientation of the original armature; you can probably get what you want by rotating only a single of these bones. And if you'd rather, you can rotate the def bones instead, it still works smoothly.
And I want to be clear that this isn't a silver bullet. You can still find positions for the IK target where the angles needed to smoothly transition between bend directions are just too large for this to handle-- usually, when the IK target is very close to the root of the chain. But adding these extra bones to loosen the angles, and to make them not care about the bend, makes a life a lot easier in typical situations.
What I need to fix is the jumping from one position to another, as without a pole target, the IK solver doesn't know in which direction to bend the rig. However, from my understanding if I use the pole target, the rotation constraints get overwritten by it and the bones rotate in all directions, as opposed to only rotating on the x axis.
