We Need to Consider Disagreement in Evaluation
| Autor: | Silviu Paun, Dirk Hovy, Tommaso Fornaciari, Alexandra Uma, Barbara Plank, Michael Fell, Massimo Poesio, Valerio Basile |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Ground truth
SOFT-LABELS Computer science Context (language use) ANNOTATION EVALUATION AGREEMENT SOFT-LABELS 02 engineering and technology ANNOTATION Data science 03 medical and health sciences 0302 clinical medicine EVALUATION AGREEMENT 030221 ophthalmology & optometry 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Adjudication |
| Zdroj: | Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future Basile, V, Fell, M, Fornaciari, T, Hovy, D, Paun, S, Plank, B, Poesio, M & Uma, A 2021, We Need to Consider Disagreement in Evaluation . in ACL-IJCNLP2021 Workshop on Benchmarking: Past, Present and Future . Association for Computational Linguistics, pp. 15-21 . https://doi.org/10.18653/v1/2021.bppf-1.3 |
| DOI: | 10.18653/v1/2021.bppf-1.3 |
| Popis: | Evaluation is of paramount importance in data- driven research fields such as Natural Language Processing (NLP) and Computer Vision (CV). But current evaluation practice in NLP, except for end-to-end tasks such as machine translation, spoken dialogue systems, or NLG, largely hinges on the existence of a single “ground truth” against which we can meaning- fully compare the prediction of a model. However, this assumption is flawed for two reasons. 1) In many cases, more than one answer is correct. 2) Even where there is a single answer, disagreement among annotators is ubiquitous, making it difficult to decide on a gold standard. We discuss three sources of disagreement: from the annotator, the data, and the con- text, and show how this affects even seemingly objective tasks. Current methods of adjudication, agreement, and evaluation ought to be re- considered at the light of this evidence. Some researchers now propose to address this issue by minimizing disagreement, creating cleaner datasets. We argue that such a simplification is likely to result in oversimplified models just as much as it would do for end-to-end tasks such as machine translation. Instead, we suggest that we need to improve today’s evaluation practice to better capture such disagreement. Datasets with multiple annotations are becoming more common, as are methods to integrate disagreement into modeling. The logical next step is to extend this to evaluation. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|