I don't know enough about math to interpret your specific math requirements but generally speaking mathematically defined models are as far as I know specified either as NURBS or in ACIS Solid format.
ACIS solids are used in many 2D CAD applications in the style of AutoCAD alikes and similar clones, but also in applications like FreeCAD or OpenSCAD.
It is generally more geared towards "hard suface" modelling, for geometric shapes like mechanical parts, engineering and industrial design.
On the other hand, NURBS is generally used for the same ends but where more organic or flowing shaped are required, like automotive industry, nautical design, aerospace engineering, aircraft design, etc.
Contrary to what you state, NURBS can be used for 3D models, while NURBS curves can be two dimensional they can also describe splines in 3D space, and NURBS surfaces are its extension to describing actual surface shapes.
Now it is not common to model human form in NURBS, most people do it using mesh based geometry mostly for a variety of practical reasons (like availability of software, animation capabilities, common tools and workflow, texturing, among others) but it has been done before.
To answer your question directly about a file format the two open standards that come to mind are either IGES or STEP, as far as open formats go that is all I know.
A better question would be What tools to use? since Blender is very poorly suited to deal with NURBS. Not only are its tools vestigial and its bare capabilities minimal at best; it cannot, as far as I know import or export any NURBS based data.
As for alternatives, the list of open source or free tools is sadly non existent, as far as I know.
The best known NURBS software is probably Rhino and CATIA, both of which are more geared towards CAD and mechanical work.
Software like Maya and 3DS Max have builtin tools for NURBS work, but it is not its main focus. They use just proprietary formats to store their data, though most can export in both IGES and STEP.
For alternatives you may look into MoI, which is a far cheaper alternative and a very capable software, more focused simplicity of workflow and designing small parts (rather than full blown projects). It is closely related to and has workflow highly compatible with Rhino.