Looking at the image above., the animation is recorded on the Empty. I assume but cannot tell for certain that all 3 rotations are animated. you only need a single rotation to be animated.
You can remove the other 2 keyframes from the axis you don't need to be animated and it will fix the issue.
The yellow color means that there are keyframes present for these properties. You should remove them. For example I assume that the rotation around the X-axis is the one you want to keep.
Press other mouse button on the Y and click on
Clear Single Keyframes. Do the same for the Z.
This will remove the keyframed animation on the Y-rotation and the Z-rotation.