CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements

Autor:	Liu, Zhu, Hu, Zhen, Liu, Ying
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	We present the results of our system for the CoMeDi Shared Task, which predicts majority votes (Subtask 1) and annotator disagreements (Subtask 2). Our approach combines model ensemble strategies with MLP-based and threshold-based methods trained on pretrained language models. Treating individual models as virtual annotators, we simulate the annotation process by designing aggregation measures that incorporate continuous similarity scores and discrete classification labels to capture both majority and disagreement. Additionally, we employ anisotropy removal techniques to enhance performance. Experimental results demonstrate the effectiveness of our methods, particularly for Subtask 2. Notably, we find that continuous similarity scores, even within the same model, align better with human disagreement patterns compared to aggregated discrete labels. Comment: 8 pages, 3 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2411.12147 Zobrazit plný text záznamu View this record from Arxiv