RadioSES: mmWave-Based Audioradio Speech Enhancement and Separation System
| Autor: | Muhammed Zahid Ozturk, Chenshu Wu, Beibei Wang, Min Wu, K. J. Ray Liu |
|---|---|
| Rok vydání: | 2023 |
| Předmět: |
Signal Processing (eess.SP)
FOS: Computer and information sciences Sound (cs.SD) Computational Mathematics Acoustics and Ultrasonics Audio and Speech Processing (eess.AS) FOS: Electrical engineering electronic engineering information engineering Computer Science (miscellaneous) Electrical Engineering and Systems Science - Signal Processing Electrical and Electronic Engineering Computer Science - Sound Electrical Engineering and Systems Science - Audio and Speech Processing |
| Zdroj: | IEEE/ACM Transactions on Audio, Speech, and Language Processing. 31:1333-1347 |
| ISSN: | 2329-9304 2329-9290 |
| DOI: | 10.1109/taslp.2023.3250846 |
| Popis: | Speech enhancement and separation have been a long-standing problem, especially with the recent advances using a single microphone. Although microphones perform well in constrained settings, their performance for speech separation decreases in noisy conditions. In this work, we propose RadioSES, an audioradio speech enhancement and separation system that overcomes inherent problems in audio-only systems. By fusing a complementary radio modality, RadioSES can estimate the number of speakers, solve source association problem, separate and enhance noisy mixture speeches, and improve both intelligibility and perceptual quality. We perform millimeter-wave sensing to detect and localize speakers, and introduce an audioradio deep learning framework to fuse the separate radio features with the mixed audio features. Extensive experiments using commercial off-the-shelf devices show that RadioSES outperforms a variety of state-of-the-art baselines, with consistent performance gains in different environmental settings. Compared with the audiovisual methods, RadioSES provides similar improvements (e.g., ~3 dB gains in SiSDR), along with the benefits of lower computational complexity and being less privacy concerning. Comment: Project webpage: https://zahidozt.github.io/RadioSES/ |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|