| Autor: |
Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin, Jianzhong Qi |
| Jazyk: |
angličtina |
| Rok vydání: |
2021 |
| Předmět: |
|
| Zdroj: |
Transactions of the Association for Computational Linguistics, Vol 9, Pp 621-640 (2021) |
| Druh dokumentu: |
article |
| ISSN: |
2307-387X |
| DOI: |
10.1162/tacl_a_00388/102842/Evaluating-Document-Coherence-Modeling |
| Popis: |
AbstractWhile pretrained language models (LMs) have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modeling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross- domain setting. |
| Databáze: |
Directory of Open Access Journals |
| Externí odkaz: |
|
