Solution: Thanks to Leander for putting up with me. (Using Blender 2.78c)
- Import the animation cleanly with my importer tool. It uses the standard blender fbx parser and cleanly imports to empties. I will provide more details soon. Its nothing special and does less than the official importer. I will provide the script to the on another question and link to it.
- Import the UE4 mesh and armature with all default settings: most important Y primary X secondary axis. Do not use joining up bones or any other features which attempt to 'fix' the bones.
- The bones will all import and they will look strange (see picture) but this is in fact the trick. They are imported as joint orientations down the x axis. They may point in the y direction but the animation is baked for x axis.
- The animation is imported with the same local rotations from the fbx file and can be used with it by adding copy rotation constraints (world to world) to every bone.
- Because the imported animation is a faithful copy the resulting animation is almost perfect (there is a twist in the upper left arm that I am working on only occurs in the last 10 frames.)[update no that's actually part of the animation in UE4, its a bug in their anim] 2
- As you can see the results are good and very faithful to the original.
- So this achieves my goal. However I would still prefer a nicer visual which isn't so confusing for the bones being just rotational joints. I honestly thought Blenders fbx import was broken until I realized that this is the intended behaviour. So in terms of development all that would be needed is a switch to change bones into a rotation bulb graphic instead of Octahedral and then the importer could default to this setting when building the armature.