So I'm using a Damped Track constraint and over that a Limit Rotation constraint as I only want the bone to rotate in X axis. This is the only setup I get it "working" but it has one tiny mistake, it rotates it a little bit to one of the sides, when it's not supposed to.
Even weirder is that if I take off the Limit Rotation constraint and I make it so the bone naturally only rotates on the axis I want then it looks fine (but I need the Limit constraint).
I would really appreciate any ideas, thanks!
Note: if I change the Convert option to anything else other than Local (or Local with Parent) then it goes all over the place.
--- EDIT ---
After doing Locked Track as constraint instead of Damped Track and Limit Rotation the rotation still happens, even when I'm not moving the tracked bone in X: