Access Audio Strips Sample data in Python for a Cross-Correlation

I want to perform an Audio Cross-Correlation in Blender (like with numpy.correlate) to align two Audio Strips. But I can't figure out how to access the Sample Data of an Audio Strip inside Blenders Python Console. How can I access the samples via Python? The Developers access these samples in the sequencer_draw.c. But with a Python Sound Object like bpy.data.scenes['Scene'].sequence_editor.sequences[0].sound I cannot access the waveform data.

I cannot use an external program like the syncaudio Add-on does as only one of my Audio Strips is based on a WAV-file, the other one is from a MTS-file and is already properly imported into an Audio Strip by Blender. And the Audaspace API as well doesn't allow me to access it.

The audio Sample Data doesn't seem to be directly accessible via python.

However, there is a workaround that suits my needs:
I can use Bake Sound to F-Curves which actually results in an already project frame rate based curve. This will add an .animation_data.action.fcurves entry for my Object I added my FCurve to with an attribute sampled_points. These sampled points have lots of FCurveSample entries that include a co Vector. Each is a pair of my frame and the corresponding sound hull curve point. Iterating this Vector I can "extract" a value for each frame which is enough to perform a cross-correlation on:

fcurves = bpy.data.objects['Empty'].animation_data.action.fcurves
samples = []
for value in fcurves[0].sampled_points.values():
samples.append(value.co.y)


Afterwards I can delete the FCurve again.

When I do this twice, I will get two sample-lists that I can feed into correlation = numpy.correlate(audio, video_strips_audio, 'valid') to perform the cross-correlation. Afterwards I can get the time shift difference between the two sound files by correlation.tolist().index(max(correlation)).

Though this doesn't work directly yet when the second sample-list starts before the first one, but I'm working on this to get it right as well... This depends on the optional mode parameter from the correlate function. I will update this answer when I get this working in general.

Why I do all this: It allows me to automatically shift two Audio files between each other until they match and can be synchronously played back even though they were recorded at the same time but their absolute start time was a bit off. So I can timewise align two or more cameras that recorded simultaneously but weren't started recording thogether. Or a Camera outputted video files audio track and an Audio Recorders Sound file.

The main advantage using this method is, that I don't need to use WAV-files only as with the mentioned add-on, but can rely on any media format including Video files that Blender can already handle via ffmpeg internally as opening a Sound file to bake to an FCurve will be handled the same way any other media/movie file I want to add to the Video Sequence Editors timeline.

• Converting any sound that blender can open to wav using blender or audacity et al, is not a huge disadvantage IMO. The "rate determining step" is baking the fcurves. Theoretically (haven't tried) converting and using syncaudio could be quicker. On the other hand having to build syncaudio from source could be troublesome for some. Apr 13 '16 at 4:08