WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context
Autor: | Mohammad Taher Pilehvar, Kiamehr Rezaee, Jose Camacho-Collados, Artem Revenko, Anna Breit |
---|---|
Rok vydání: | 2021 |
Předmět: |
FOS: Computer and information sciences
Computer Science - Computation and Language Computer science business.industry Context (language use) Machine learning computer.software_genre Task (project management) Set (abstract data type) Binary classification Benchmark (computing) Uniqueness Language model Artificial intelligence Baseline (configuration management) business Computation and Language (cs.CL) computer |
Zdroj: | EACL Scopus-Elsevier |
Popis: | We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disambiguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its uniqueness in the formulation as a binary classification task thus being independent of external sense inventories, and the coverage of various domains. This makes the dataset highly flexible for the evaluation of a diverse set of models and systems in and across domains. WiC-TSV provides three different evaluation settings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in out-of-domain settings. WiC-TSV data is available at https://competitions.codalab.org/competitions/23683 Accepted to EACL 2021. Reference paper of the SemDeep WiC-TSV challenge: https://competitions.codalab.org/competitions/23683 |
Databáze: | OpenAIRE |
Externí odkaz: |