In the game engine I'm trying to make the different types of enemies pass through each other, and only interact with either the player or the player's weapon. I've made them rigid body objects, and tried adding them to different collision groups but that doesn't seem to do anything.
Collision groups are the correct way to do this.
Set your "bad guys'" collision group and collision mask up like this.
Set the "good guy" up like this.
Now here is why. The Collision group is conceptually assigning a number to the object say 1-8. Your good guy has collision group one. The Collision Mask is what number collision group it can collide with.
So looking at your bad guy, it can only collide with other objects that are assigned to collision group 1. Itself is set to collision group two so it will not hit other bad guys. Your good guy is set to collision group one, and it can collides with all groups.
Blend file with bge collision demo.
You can do this with the collision masks. Add the enemies to one collision group and the player and its weapon to another group. Then, in the collision masks for the enemies, deselect the groups with which you don't want them to collide.
I tried to make animated gifs, but the screencast wouldn't record the "game" when I played it out. However, as you can see in the image above, the sphere doesn't have the group it belongs to in its mask. The plane is setup the exact same way. And when I play it out, the sphere falls right through the plane. When I change the masks to include the group the sphere and the plane belong to, the sphere lands on the plane.