Back around 2005, fresh out of college with a physics and neuroscience degree, I was actually working on a project similar to this, which would take a synthesized sound and construct a rough model of the physical instrument necessary to create it. I learned a few things back then about the digital construction of sounds from sculpted materials to be able to add a little to this question.
The first thing you would need to concern yourself with is how precise you would want the sound to be, as there are a lot of factors involved. What kind of medium (precisely) is the event occurring in? What are the active elements made of? Are they consistently made of the same material, or is there buffering internally? How flexible is this material? Has it been scored or warped in any way? What temperature and phase is it at?
I could go on; but all of these things have a significant effect on acoustics (as anyone who has ever had to refurbish a broken musical instrument knows). The good news is, if we're assuming simple and consistent materials, mechanical engineers have mapped out just about all of them, which will reasonably easily fit into an inline table on the LTS of most modern machines.
After that, you have the precision of the sound. We're talking about something that ultimately involves every molecule of the material, and if it isn't a perfectly crystalline material (which just doesn't happen in the real world), then each molecule is going to behave a little differently. I sincerely doubt that you're going to need an exact match, but my point is that you will need a "good enough" level to stop at. Think of it as a number of samples, but for safety's sake, throw it to around 100,000 or more. Sounds audible to a human being are generally between 20 Hz and 20 KHz, which means by the Nyquist theorem you would need about double that to render them at all, and quite a few times more if you don't want significant amplitude distortion.
But, I have more good news. Most of the work has been done by electronic musicians, going back to Lev Sergeyevich Termen in 1930. You're familiar, I'm sure, with square, triangle, saw, and sine waves? They are in common usage because they closely emulate the acoustic behavior of orchestral instruments. Take any dirt-cheap 80s keyboard (and even the expensive ones) and you'll find that every MIDI program on it consists of a combination of these waves, providing us with expansive muscial versatility; particularly when you apply the same principles on modern hardware.
Additionally, Homer Dudley applied his knowledge of the physical structure and deformation of human vocal cords and created a device in 1937 capable of constructing detailed human-like speech from carrier waves. He called it a "Vocal Decoder", or "Vocoder", and it was effectively a complicated organ the size of a small barn, but produced recognizable sentences in the hands of a trained operator. It debuted at the World's Fair. I'm sure he was thinking very much about the 1930s equivalent of this notion of building sounds from modeled objects. Today, of course, the technology is still around; but it's embedded in microprocessors or purely digital DAWs.
(Note that I am not referring to vocoders as you'll find as synthesizers in any good DAW; nor am I referring to those used in cell phone towers; though the technologies are tightly related.)
The down side is that, human-like or not, look up a recording of Dudley's vocoder and you'll immediately feel the artificiality of it. (Be fair, it was one of the world's first synthesizers.)
You might look up acoustic work by both of them to get started.
So, knowing what we do of the Fourier decomposition of one sound into a collection of component sounds, and the usage of carrier waves to establish complicated (but known) physical distortions; and assuming you're willing to cut corners here and there don't exactly have the time or money for a personal Google Deep Mind, I would say that your best bet is this.
Take known rigidbody (or softbody) shapes, and map their momentum and tensile strength to an octree space partition. Fill the void with the tensile properties of the medium they're immersed in (say, air, at room temperature and atmospheric pressure, unless you're looking for something interesting). For boundaries, assume an infinite expanse of a select physical property. During your animation (which should be high FPS!) trace the compression and restoration of each material, at a bare minimum of 80KHz or higher for recognizable sound. (That's 80,000 FPS.) Ray cast to the listener (probably your camera), and consider power loss with distance, and any distortions that the medium might put on it. Lastly, if it still doesn't sound precise enough, up the number of partitions in the octree, or fit it to the actual shape of your objects a little better.
You have your basic framework for forming sound, which can easily be done with Python. However, you shouldn't be surprised if you break 1,000,000 FPS for the audio before it sounds "right" to you. If we're being exact, this is not computationally cheap. (You will almost certainly need to rent time on a server farm.)
Another established approach is from Google, with their WaveNet spin off of Deep Mind. You might take a look at it; it isn't precisely related to 3D modeling, but they use a technique where a voice is established by considering each sample frame's most likely effect on adjacent sample frames and feeding them through a neural network. It's sort of the next step beyond Concatenative or Parametric Voice Synthesis. Unfortunately, like waiting on a Cycles bake of an extraordinarily complicated scene, it can take months to complete and isn't short on computing power needs; it also considers sound independently from what's creating it.
It may inspire you, though; as a scaled down method could combine the pre-recorded sounds of similar objects to your manifolds in a rational way to create a new one that best fits the bill. It would be much less demanding on computing power, but potentially might require a larger and more flexible library of data.