I am not sure what is this called, but maybe similar to motion capture. For example I want to animate a hand in Blender. Instead of motion capturing by pointing a camera at my hand and it tracks in 2D, I hold my phone and set the forward direction then I can move a point in blender as if my phone is that point.
Another case I would like to use something like this is camera animation. If I can hold my phone, set a starting position, and Blender's camera can move along with my phone I could record an animation to make panning or just holding a camera still looks more human. (with little noises and all)
If it does not exist in Blender but exist in other programs I would like to know a name of it. Or if this is not a common idea I might learn to contribute to Blender one day. (but first I don't want to reinvent the wheel if there is any) Newest iOS 11 has ARKit, it might have some interesting pre-made phone localizing function without using markers and I might try it.