After having my camera tracked, I decided to put in a cube for testing. But then I've noticed this strange artifact where every object kinda flickers and changes it's rotation.
You can clearly see the tracker shifting on the video posted on the question, and it happens at the moment where the reconstruction fails.
Successful reconstruction can only be achieved with accurate tracking
You should aim to have a solve error of less than 0.3 pixels.
There are plenty of tools to help you get more accurate tracking.