Global Autoregressive Models for Data-Efficient Sequence Learning
Autor: | Marc Dymetman, Jean-Marc Andreoli, Tetiana Parshakova |
---|---|
Jazyk: | angličtina |
Rok vydání: | 2019 |
Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Computer science Computer Science - Artificial Intelligence Inference Machine Learning (stat.ML) Perplexity reduction 010501 environmental sciences 01 natural sciences Machine Learning (cs.LG) Statistics - Machine Learning Component (UML) 0502 economics and business Statistics::Methodology 050207 economics 0105 earth and related environmental sciences Computer Science - Computation and Language Statistics::Applications business.industry 05 social sciences Statistics::Computation Artificial Intelligence (cs.AI) Distribution (mathematics) Autoregressive model A priori and a posteriori Sequence learning Artificial intelligence business Focus (optics) Computation and Language (cs.CL) Algorithm |
Zdroj: | CoNLL |
Popis: | Standard autoregressive seq2seq models are easily trained by max-likelihood, but tend to show poor results under small-data conditions. We introduce a class of seq2seq models, GAMs (Global Autoregressive Models), which combine an autoregressive component with a log-linear component, allowing the use of global \textit{a priori} features to compensate for lack of data. We train these models in two steps. In the first step, we obtain an \emph{unnormalized} GAM that maximizes the likelihood of the data, but is improper for fast inference or evaluation. In the second step, we use this GAM to train (by distillation) a second autoregressive model that approximates the \emph{normalized} distribution associated with the GAM, and can be used for fast inference and evaluation. Our experiments focus on language modelling under synthetic conditions and show a strong perplexity reduction of using the second autoregressive model over the standard one. To appear in CONLL (The SIGNLL Conference on Computational Natural Language Learning) Hong Kong, Nov. 2019 |
Databáze: | OpenAIRE |
Externí odkaz: |