Aligning Orchestral Music

Below we show how to use this library to align some of the audio used in the paper. For this notebook to work, we assume the user has installed the audio requirements (pip install -r requirements_audio.txt at the root of the repository). we assume that the user has already downloaded the “short” orchestral pieces in the benchmark using the script in experiments/orchestral.py. We do not provide the pieces here for copyright reasons.

Audio example 1: Vivalid’s Spring

The first step is to load in the audio

[1]:
import linmdtw
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook
import warnings
warnings.filterwarnings("ignore")
import IPython.display as ipd

sr = 44100
x0_0, sr = linmdtw.load_audio("../experiments/OrchestralPieces/Short/0_0.mp3", sr)
x0_1, sr = linmdtw.load_audio("../experiments/OrchestralPieces/Short/0_1.mp3", sr)

Next, we’ll compute the “MFCC mod” features for each audio clip, as described in [1]

[1] Gadermaier, Thassilo, and Gerhard Widmer. “A Study of Annotation and Alignment Accuracy for Performance Comparison in Complex Orchestral Music.” arXiv preprint arXiv:1910.07394 (2019).

[2]:
hop_length = 512
X0_0 = linmdtw.get_mfcc_mod(x0_0, sr, hop_length)
X0_1 = linmdtw.get_mfcc_mod(x0_1, sr, hop_length)

Now, we can extract a warping path between the two audio streams using the main DTW library

[3]:
import time
metadata = {'totalCells':0, 'M':X0_0.shape[0], 'N':X0_1.shape[0],
            'timeStart':time.time(), 'perc':10}
path0 = linmdtw.linmdtw(X0_0, X0_1, do_gpu=True, metadata=metadata)[1]
Parallel Alignment 10.0% Elapsed time: 6.77
Parallel Alignment 20.0% Elapsed time: 13.5
Parallel Alignment 30.0% Elapsed time: 20.3
Parallel Alignment 40.0% Elapsed time: 27
Parallel Alignment 50.0% Elapsed time: 33.9
Parallel Alignment 60.0% Elapsed time: 40.7
Parallel Alignment 70.0% Elapsed time: 48.3
Parallel Alignment 80.0% Elapsed time: 55.8
Parallel Alignment 90.0% Elapsed time: 62.7

Before we apply the computed warping path, let’s compare the first 40 seconds of the two audio clips side by side. We’ll put the first one in the left ear and the second one in the right ear. The one on the left goes faster than the one on the right, but it starts later. Because of this, they are in sync for a brief moment, but the left one then overtakes the right one for the rest of it.

[4]:
xunsync0 = np.zeros((sr*40, 2))
xunsync0[:, 0] = x0_0[0:sr*40]
xunsync0[:, 1] = x0_1[0:sr*40]
linmdtw.save_audio(xunsync0, sr, "unsync0")
ipd.Audio("unsync0.mp3")
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'unsync0.wav':
  Duration: 00:00:40.00, bitrate: 5644 kb/s
    Stream #0:0: Audio: pcm_f64le ([3][0][0][0] / 0x0003), 44100 Hz, stereo, dbl, 5644 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_f64le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to 'unsync0.mp3':
  Metadata:
    TSSE            : Lavf58.29.100
    Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.54.100 libmp3lame
size=     626kB time=00:00:40.02 bitrate= 128.1kbits/s speed=50.7x
video:0kB audio:626kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.039486%
[4]:

Let’s now apply the computed warping path to see how the alignment went. This library wraps arround the pyrubberband library, which we can use to stretch the audio in x1 to match x2, according to this warping path. The method stretch_audio returns a stereo audio stream with the resulting stretched version of x1 in the left ear and the original version of x2 in the right ear. Let’s save the first 30 seconds of the aligned audio to disk and listen to it

[5]:
xsync0 = linmdtw.stretch_audio(x0_0, x0_1, sr, path0, hop_length)
linmdtw.save_audio(xsync0[0:sr*30, ::], sr, "sync0")
ipd.Audio("sync0.mp3")
Stretching...
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'sync0.wav':
  Duration: 00:00:30.00, bitrate: 5644 kb/s
    Stream #0:0: Audio: pcm_f64le ([3][0][0][0] / 0x0003), 44100 Hz, stereo, dbl, 5644 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_f64le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to 'sync0.mp3':
  Metadata:
    TSSE            : Lavf58.29.100
    Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.54.100 libmp3lame
size=     470kB time=00:00:30.01 bitrate= 128.2kbits/s speed=49.3x
video:0kB audio:469kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.052637%
[5]: