Using Fluid Control objects can significantly affect your bake times - particularly when using a large Radius in the control object.
The control object 'radius' determines how close fluid needs to get to the control object before it is influenced by it. When fluid is being influenced by a control object the fluid simulation has to do a lot more work to determine how that fluid is affected by the control object (which effectively emits a force on the fluid). Therefore, the more fluid is within the radius of the control object, the longer the simulation will take.
This means that at the start of the animation the baking may take a small amount of time but as fluid gets closer to the control objects it will drastically slow down.
To limit the overhead of control objects, keep the Radius of effect as small as possible for your purposes. If you are only using stationary control objects you may consider reducing the Velocity Force radius to zero.