Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Autor:	Parker, Julian D, Smirnov, Anton, Pons, Jordi, Carr, CJ, Zukowski, Zack, Evans, Zach, Liu, Xubo
Rok vydání:	2024
Předmět:	Electrical Engineering and Systems Science - Audio and Speech Processing Computer Science - Artificial Intelligence Computer Science - Machine Learning Computer Science - Sound Electrical Engineering and Systems Science - Signal Processing
Druh dokumentu:	Working Paper
Popis:	The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models have concentrated on low parameter-count architectures using only components with strong inductive biases. In this work we show that by scaling a transformer architecture with large parameter count to this problem, and applying a flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to reach state-of-the-art speech quality at extremely low bit-rates of $400$ or $700$ bits-per-second. The trained models strongly out-perform existing baselines in both objective and subjective tests.
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2411.19842 Zobrazit plný text záznamu View this record from Arxiv