Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning
| Autor: | Dirk Hovy, Tommaso Fornaciari, Barbara Plank, Silviu Paun, Alexandra Uma, Massimo Poesio |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Computer science
Multi-task learning 02 engineering and technology 010501 environmental sciences Overfitting Machine learning computer.software_genre 01 natural sciences MULTI-TASK LEARNING Task (project management) 0202 electrical engineering electronic engineering information engineering Divergence (statistics) SOFT-LABELS MULTI-TASK LEARNING AGREEMENT 0105 earth and related environmental sciences Ground truth SOFT-LABELS Artificial neural network business.industry Supervised learning ComputingMethodologies_PATTERNRECOGNITION AGREEMENT Probability distribution 020201 artificial intelligence & image processing Artificial intelligence business computer |
| Zdroj: | Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Fornaciari, T, Uma, A, Paun, S, Plank, B, Hovy, D & Poesio, M 2021, Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning . in Proceedings of NAACL . Association for Computational Linguistics, pp. 2591–2597 . NAACL-HLT |
| DOI: | 10.18653/v1/2021.naacl-main.204 |
| Popis: | Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a novel method to incorporate this disagreement as information: in addition to the standard error computation, we use soft-labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural network. We measure the divergence between the predictions and the target soft-labels with several loss-functions and evaluate the models on various NLP tasks. We find that the soft-label prediction auxiliary task reduces the penalty for errors on ambiguous entities, and thereby mitigates overfitting. It significantly improves performance across tasks, beyond the standard approach and prior work. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|