Let's say I have a character with four arms, all part of the same armature. Is there a way I can animate one arm, turn this into an NLA strip and use them on the other bones or do the other arms have to be separate armatures?
You would need to create actions for all of the four arms.
Reusing one action for different bones wouldn't work because the action holds the keyframed properties like rotation and location only for the bones selected while adding a keyframe.
But you can try to copy Ctrl-C and paste Ctrl-V the selected property values (or f-curves) from one action to another, this requires that you insert an initial keyframe for the bone to animate.