Self-Normalization Properties of Language Modeling

Autor:	Goldberger, Jacob, Melamud, Oren
Rok vydání:	2018
Předmět:	Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	Self-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. In the context of language modeling, this property is particularly appealing as it may significantly reduce run-times due to large word vocabularies. In this study, we provide a comprehensive investigation of language modeling self-normalization. First, we theoretically analyze the inherent self-normalization properties of Noise Contrastive Estimation (NCE) language models. Then, we compare them empirically to softmax-based approaches, which are self-normalized using explicit regularization, and suggest a hybrid model with compelling properties. Finally, we uncover a surprising negative correlation between self-normalization and perplexity across the board, as well as some regularity in the observed errors, which may potentially be used for improving self-normalization algorithms in the future. Comment: Accepted to Coling 2018
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1806.00913 Zobrazit plný text záznamu View this record from Arxiv