Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks
| Autor: | Oortwijn, Y., Ossenkoppele, T., Betti, A., Belz, A., Agarwal, S., Graham, Y., Reiter, E., Shimorina, A. |
|---|---|
| Přispěvatelé: | ILLC (FGw), Logic and Language (ILLC, FNWI/FGw), ILLC (FNWI/FGw) |
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Zdroj: | Human Evaluation of NLP Systems (HumEval): EACL 2021 : proceedings of the workshop : April 19, 2021 Human Evaluation of NLP Systems (HumEval) |
| Popis: | We present a systematic procedure for interrater disagreement resolution. The procedure is general, but of particular use in multiple-annotator tasks geared towards ground truth construction. We motivate our proposal by arguing that, barring cases in which the researchers’ goal is to elicit different viewpoints, interrater disagreement is a sign of poor quality in the design or the description of a task. Consensus among annotators, we maintain, should be striven for, through a systematic procedure for disagreement resolution such as the one we describe. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|