Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models
| Autor: | Lombardo, Pierangelo, Boiardi, Alessio, Colombo, Luca, Schiavone, Angelo, Tamagnone, Nicolò |
|---|---|
| Rok vydání: | 2020 |
| Předmět: | |
| Zdroj: | Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 3081-3093) |
| Druh dokumentu: | Working Paper |
| Popis: | The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks. In this work, we give a threefold contribution to address these requirements: (i) we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatedness-based evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. Finally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol. Comment: This is a pre-print of an article published in the proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) |
| Databáze: | arXiv |
| Externí odkaz: |
|