Discrete representations in neural models of spoken language

Autor:	Higy, Bertrand, Gelderloos, Lieke, Alishahi, Afra, Chrupała, Grzegorz
Rok vydání:	2021
Předmět:	Computer Science - Computation and Language Computer Science - Machine Learning Computer Science - Sound Electrical Engineering and Systems Science - Audio and Speech Processing
Druh dokumentu:	Working Paper
Popis:	The distributed and continuous representations used by neural networks are at odds with representations employed in linguistics, which are typically symbolic. Vector quantization has been proposed as a way to induce discrete neural representations that are closer in nature to their linguistic counterparts. However, it is not clear which metrics are the best-suited to analyze such discrete representations. We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We compare the results they show when applied to two different models, while systematically studying the effect of the placement and size of the discretization layer. We find that different evaluation regimes can give inconsistent results. While we can attribute them to the properties of the different metrics in most cases, one point of concern remains: the use of minimal pairs of phoneme triples as stimuli disadvantages larger discrete unit inventories, unlike metrics applied to complete utterances. Furthermore, while in general vector quantization induces representations that correlate with units posited in linguistics, the strength of this correlation is only moderate. Comment: Accepted for publication at BlackboxNLP 2021
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2105.05582 Zobrazit plný text záznamu View this record from Arxiv