Scaling and Disagreements: Bias, Noise, and Ambiguity

Autor: Alexandra Uma, Dina Almanea, Massimo Poesio

Publikováno v: Frontiers in Artificial Intelligence, Vol 5 (2022)

Crowdsourced data are often rife with disagreement, either because of genuine item ambiguity, overlapping labels, subjectivity, or annotator error. Hence, a variety of methods have been developed for learning from data containing disagreement. One of

Externí odkaz: https://doaj.org/article/e87afaa098da4f31a748312dc42dbc2f

Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning

Autor: Dirk Hovy, Tommaso Fornaciari, Barbara Plank, Silviu Paun, Alexandra Uma, Massimo Poesio

Publikováno v: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Fornaciari, T, Uma, A, Paun, S, Plank, B, Hovy, D & Poesio, M 2021, Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning . in Proceedings of NAACL . Association for Computational Linguistics, pp. 2591–2597 .
NAACL-HLT

Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a n

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::86a09e55737fbf08b689aa3eb596b558

We Need to Consider Disagreement in Evaluation

Autor: Silviu Paun, Dirk Hovy, Tommaso Fornaciari, Alexandra Uma, Barbara Plank, Michael Fell, Massimo Poesio, Valerio Basile

Publikováno v: Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future
Basile, V, Fell, M, Fornaciari, T, Hovy, D, Paun, S, Plank, B, Poesio, M & Uma, A 2021, We Need to Consider Disagreement in Evaluation . in ACL-IJCNLP2021 Workshop on Benchmarking: Past, Present and Future . Association for Computational Linguistics, pp. 15-21 . https://doi.org/10.18653/v1/2021.bppf-1.3

Evaluation is of paramount importance in data- driven research fields such as Natural Language Processing (NLP) and Computer Vision (CV). But current evaluation practice in NLP, except for end-to-end tasks such as machine translation, spoken dialogue

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::bf9f871b2d3b1d1cfeec4726cd45ab4e
https://doi.org/10.18653/v1/2021.bppf-1.3

Anaphora Resolution with the ARRAU Corpus

Autor: Heike Zinsmeister, Juntao Yu, Yulia Grishina, Fabian Simonjetz, Nafise Sadat Moosavi, Olga Uryupina, Adam Roussel, Varada Kolhatkar, Alexandra Uma, Massimo Poesio, Ina Roesiger

Publikováno v: Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference

The ARRAU corpus is an anaphorically annotated corpus of English providing rich linguistic information about anaphora resolution. The most distinctive feature of the corpus is the annotation of a wide range of anaphoric relations, including bridging

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::dbb87fb54d5f90fae5eb4f3ed1e20a2a