Leveraging Pre-trained AudioLDM for Text to Sound Generation: A Benchmark Study
Autor: | Yuan, Yi, Liu, Haohe, Liang, Jinhua, Liu, Xubo, Plumbley, Mark D., Wang, Wenwu |
---|---|
Rok vydání: | 2023 |
Předmět: |
FOS: Computer and information sciences
Sound (cs.SD) Artificial Intelligence (cs.AI) Computer Science - Artificial Intelligence Audio and Speech Processing (eess.AS) FOS: Electrical engineering electronic engineering information engineering Computer Science - Sound Computer Science - Multimedia Electrical Engineering and Systems Science - Audio and Speech Processing Multimedia (cs.MM) |
DOI: | 10.48550/arxiv.2303.03857 |
Popis: | Deep neural networks have recently achieved breakthroughs in sound generation with text prompts. Despite their promising performance, current text-to-sound generation models face issues on small-scale datasets (e.g., overfitting), significantly limiting their performance. In this paper, we investigate the use of pre-trained AudioLDM, the state-of-the-art model for text-to-audio generation, as the backbone for sound generation. Our study demonstrates the advantages of using pre-trained models for text-to-sound generation, especially in data-scarcity scenarios. In addition, experiments show that different training strategies (e.g., training conditions) may affect the performance of AudioLDM on datasets of different scales. To facilitate future studies, we also evaluate various text-to-sound generation systems on several frequently used datasets under the same evaluation protocols, which allow fair comparisons and benchmarking of these methods on the common ground. Comment: EUSIPCO 2023 |
Databáze: | OpenAIRE |
Externí odkaz: |