8
$\begingroup$

I know this sounds like a ridiculous/newbie question, and I know it's been asked before (and yes, I am currently aware that there's no 'official' method for doing this), but is it theoretically possible to obtain data (from, say for example, the collision of objects) and through some elaborate scripting, be able to actually simulate what it would physically "sound" like? And if it were possible, how could it be done?

I can imagine that the adjustment of objects' positions relative to the active camera would be a fairly simple and straightforward place to start out (rapid oscillations based on the locations of each object could simulate sound wave pressures and be delivered straight to headphones), but other sounds appear a bit more complicated because of the way the physics system is already set up, with an emphasis on speed & efficiency rather than complete and total accuracy.

Anyways, I guess my primary question is: how would one go about beginning to script such a thing? Have there been attempts to do this already? Has anyone considered implementing an idea like this to create physically accurate sounds? (rather than taking a sound effect that has no relation to the physical qualities of the object it's ascribed to and using it for that purpose)

$\endgroup$
  • $\begingroup$ I am sure I have seen some siggrpah papers about this kind of research. It is not a trivial job to accomplish this. $\endgroup$ – satishgoda Jun 8 '17 at 8:57
  • $\begingroup$ Somebody just needs to write a render engine for audio acoustics. Blender can provide the geometry. This would of course be a massive project, although really cool. Alternatively, if you're not looking for acoustic realism but just looking to to get interesting effects, you could experiment with converting render result color data into audio frequencies. $\endgroup$ – Mentalist Jun 8 '17 at 16:22
  • $\begingroup$ I've asked myself the same question and I think I even saw a similar question on here somewhere. I think it would be very difficult or maybe even another program altogether. Making objects break (for example) in Blender is already hard enough (compared to, say, Houdini)... also meshes are usually empty - would they sound different depending on material? What should a material which is 50% glass and glossy behave/sound like? $\endgroup$ – MicroMachine Jun 9 '17 at 1:07
  • 1
    $\begingroup$ @MicroMachine Similar to 3D printing, each mesh would need to have a volume in order to be calculated correctly. As for materials, shaders would be irrelevant. there would need to be new material properties coded within the engine for handling sound - like shaders, but for audio. These would tell the render engine, for example, "this object is made of steel" and "this other object is made of wood". Similar to how with IOR each kind of material has its own refraction value, acoustic materials would need to have their own values that define their density and how they resonate. $\endgroup$ – Mentalist Jun 9 '17 at 21:10
  • 1
    $\begingroup$ I just came across a very relevant and interesting video on this subject so I thought I'd share it: youtube.com/watch?v=DzsZ2qMtEUE $\endgroup$ – Mentalist Jun 16 '17 at 16:15
5
$\begingroup$

There is active research happening in this field currently and Stanford is in the forefront of this research. They presented their work at SIGGRAPH, 2018. I hope soon it can be integrated to Blenderv2.8. Check out their work on the same -

Paper - https://graphics.stanford.edu/projects/wavesolver/

Video Explanation - https://www.youtube.com/watch?v=2mjPxcE8UDg

Articles - https://engineering.stanford.edu/magazine/article/crash-bam-plop-new-system-brings-sounds-animation-life

$\endgroup$
0
$\begingroup$

Back around 2005, fresh out of college with a physics and neuroscience degree, I was actually working on a project similar to this, which would take a synthesized sound and construct a rough model of the physical instrument necessary to create it. I learned a few things back then about the digital construction of sounds from sculpted materials to be able to add a little to this question.

PHYSICS

The first thing you would need to concern yourself with is how precise you would want the sound to be, as there are a lot of factors involved. What kind of medium (precisely) is the event occurring in? What are the active elements made of? Are they consistently made of the same material, or is there buffering internally? How flexible is this material? Has it been scored or warped in any way? What temperature and phase is it at?

I could go on; but all of these things have a significant effect on acoustics (as anyone who has ever had to refurbish a broken musical instrument knows). The good news is, if we're assuming simple and consistent materials, mechanical engineers have mapped out just about all of them, which will reasonably easily fit into an inline table on the LTS of most modern machines.

After that, you have the precision of the sound. We're talking about something that ultimately involves every molecule of the material, and if it isn't a perfectly crystalline material (which just doesn't happen in the real world), then each molecule is going to behave a little differently. I sincerely doubt that you're going to need an exact match, but my point is that you will need a "good enough" level to stop at. Think of it as a number of samples, but for safety's sake, throw it to around 100,000 or more. Sounds audible to a human being are generally between 20 Hz and 20 KHz, which means by the Nyquist theorem you would need about double that to render them at all, and quite a few times more if you don't want significant amplitude distortion.

HISTORY

But, I have more good news. Most of the work has been done by electronic musicians, going back to Lev Sergeyevich Termen in 1930. You're familiar, I'm sure, with square, triangle, saw, and sine waves? They are in common usage because they closely emulate the acoustic behavior of orchestral instruments. Take any dirt-cheap 80s keyboard (and even the expensive ones) and you'll find that every MIDI program on it consists of a combination of these waves, providing us with expansive muscial versatility; particularly when you apply the same principles on modern hardware.

Additionally, Homer Dudley applied his knowledge of the physical structure and deformation of human vocal cords and created a device in 1937 capable of constructing detailed human-like speech from carrier waves. He called it a "Vocal Decoder", or "Vocoder", and it was effectively a complicated organ the size of a small barn, but produced recognizable sentences in the hands of a trained operator. It debuted at the World's Fair. I'm sure he was thinking very much about the 1930s equivalent of this notion of building sounds from modeled objects. Today, of course, the technology is still around; but it's embedded in microprocessors or purely digital DAWs.

(Note that I am not referring to vocoders as you'll find as synthesizers in any good DAW; nor am I referring to those used in cell phone towers; though the technologies are tightly related.)

The down side is that, human-like or not, look up a recording of Dudley's vocoder and you'll immediately feel the artificiality of it. (Be fair, it was one of the world's first synthesizers.)

You might look up acoustic work by both of them to get started.

APPLICATION

So, knowing what we do of the Fourier decomposition of one sound into a collection of component sounds, and the usage of carrier waves to establish complicated (but known) physical distortions; and assuming you're willing to cut corners here and there don't exactly have the time or money for a personal Google Deep Mind, I would say that your best bet is this.

Take known rigidbody (or softbody) shapes, and map their momentum and tensile strength to an octree space partition. Fill the void with the tensile properties of the medium they're immersed in (say, air, at room temperature and atmospheric pressure, unless you're looking for something interesting). For boundaries, assume an infinite expanse of a select physical property. During your animation (which should be high FPS!) trace the compression and restoration of each material, at a bare minimum of 80KHz or higher for recognizable sound. (That's 80,000 FPS.) Ray cast to the listener (probably your camera), and consider power loss with distance, and any distortions that the medium might put on it. Lastly, if it still doesn't sound precise enough, up the number of partitions in the octree, or fit it to the actual shape of your objects a little better.

You have your basic framework for forming sound, which can easily be done with Python. However, you shouldn't be surprised if you break 1,000,000 FPS for the audio before it sounds "right" to you. If we're being exact, this is not computationally cheap. (You will almost certainly need to rent time on a server farm.)

Another established approach is from Google, with their WaveNet spin off of Deep Mind. You might take a look at it; it isn't precisely related to 3D modeling, but they use a technique where a voice is established by considering each sample frame's most likely effect on adjacent sample frames and feeding them through a neural network. It's sort of the next step beyond Concatenative or Parametric Voice Synthesis. Unfortunately, like waiting on a Cycles bake of an extraordinarily complicated scene, it can take months to complete and isn't short on computing power needs; it also considers sound independently from what's creating it.

It may inspire you, though; as a scaled down method could combine the pre-recorded sounds of similar objects to your manifolds in a rational way to create a new one that best fits the bill. It would be much less demanding on computing power, but potentially might require a larger and more flexible library of data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.