Generalising Kendall's Tau for Noisy and Incomplete Preference Judgements
| Autor: | Riku Togashi, Tetsuya Sakai |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Computer science
business.industry User modeling InformationSystems_INFORMATIONSTORAGEANDRETRIEVAL 05 social sciences Probabilistic logic ComputerApplications_COMPUTERSINOTHERSYSTEMS 02 engineering and technology computer.software_genre Crowdsourcing Measure (mathematics) Preference Ranking (information retrieval) 020204 information systems 0202 electrical engineering electronic engineering information engineering Relevance (information retrieval) Artificial intelligence 0509 other social sciences 050904 information & library sciences business computer Natural language processing Rank correlation |
| Zdroj: | ICTIR |
| DOI: | 10.1145/3341981.3344246 |
| Popis: | We propose a new ranking evaluation measure for situations where multiple preference judgements are given for each item pair but they may be noisy (i.e., some judgements are unreliable) and/or incomplete (i.e., some judgements are missing). While it is generally easier for assessors to conduct preference judgements than absolute judgements, it is often not practical to obtain preference judgements for all combinations of documents. However, this problem can be overcome if we can effectively utilise noisy and incomplete preference judgements such as those that can be obtained from crowdsourcing. Our measure, η, is based on a simple probabilistic user model of the labellers which assumes that each document is associated with a graded relevance score for a given query. We also consider situations where multiple preference probabilities, rather than preference labels, are given for each document pair. For example, in the absence of manual preference judgements, one might want to employ an ensemble of machine learning techniques to obtain such estimated probabilities. For this scenario, we propose another ranking evaluation measure called η_p $. Through simulated experiments, we demonstrate that our proposed measures η and η_p$ can evaluate rankings more reliably than τ\mbox- b$, a popular rank correlation measure. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|