Universal Sound Separation
Ilya Kavalerov1,2 | Scott Wisdom1 | Hakan Erdogan1 | Brian Patton1 | Kevin Wilson1 | Jonathan Le Roux3 | John R. Hershey1 |
1Google Research |
2University of Maryland |
3Mitsubishi Electric Research Labs |
Abstract
Recent deep learning approaches have achieved impressive performance on speech enhancement and separation tasks. However, these approaches have not been investigated for separating mixtures of arbitrary sounds of different types, a task we refer to as universal sound separation, and it is unknown whether performance on speech tasks carries over to non-speech tasks. To study this question, we develop a universal dataset of mixtures containing arbitrary sounds, and use it to investigate the space of mask-based separation architectures, varying both the overall network architecture and the framewise analysis-synthesis basis for signal transformations. These network architectures include convolutional long short-term
memory networks and time-dilated convolution stacks inspired
by the recent success of time-domain enhancement networks like
ConvTasNet. For the latter architecture, we also propose novel
modifications that further improve separation performance. In
terms of the framewise analysis-synthesis basis, we explore using
either a short-time Fourier transform (STFT) or a learnable basis,
as used in ConvTasNet, and for both of these bases, we examine
the effect of window size. In particular, for STFTs, we find that
longer windows (25-50 ms) work best for speech/non-speech separation, while shorter windows (2.5 ms) work best for arbitrary
sounds. For learnable bases, shorter windows (2.5 ms) work best
on all tasks. Surprisingly, for universal sound separation, STFTs
outperform learnable bases. Our best methods produce an improvement in scale-invariant signal-to-distortion ratio of over 13 dB
for speech/non-speech separation and close to 10 dB for universal
sound separation.
Paper
"Universal Sound Separation", [PDF] |
Audio Demos
Speech Enhancement | 2-Source Universal Sound Separation | 3-Source Universal Sound Separation |
Dataset Recipe
Coming soon. |
Last updated: Aug 2019