I have two seperate armatures I want to connect to eachother, CTRL-J adds the first armature to the second and the mesh still works as expected. But when I move the bones on one armature the bones on the other armature won't follow. So how could I get the bones to follow? The model is properly weight painted.
After you joined the two meshes, you must select in edit mode all the bones of the second armature and parent them to the "master" bone of the first armature, the one which moves all the others (choose "keep offset" option when asked to).