The Importance of Context in Very Low Resource Language Modeling
| Autor: | Edman, Lukas, Toral Ruiz, Antonio, van Noord, Gertjan, Bandyopadhyay, Sivaji, Devi, Sobha Lalitha, Bhattacharyya, Pushpak |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Zdroj: | Proceedings of the 18th International Conference on Natural Language Processing (ICON), 86-92 STARTPAGE=86;ENDPAGE=92;TITLE=Proceedings of the 18th International Conference on Natural Language Processing (ICON) |
| Popis: | This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|