Building Test Collections using Bandit Techniques: A Reproducibility Study
| Autor: | Bahadir Altun, Mucahid Kutlu |
|---|---|
| Přispěvatelé: | TOBB ETU, Faculty of Engineering, Department of Computer Engineering, TOBB ETÜ, Mühendislik Fakültesi, Bilgisayar Mühendisliği Bölümü, Kutlu, Mücahid |
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
Computer science
Pooling 02 engineering and technology Machine learning computer.software_genre GeneralLiterature_MISCELLANEOUS Set (abstract data type) 020204 information systems 0202 electrical engineering electronic engineering information engineering test collections information retrieval Reliability (statistics) Reusability Reproducibility evaluation business.industry 05 social sciences Intelligent document Test (assessment) Bandit methods Selection method Artificial intelligence 0509 other social sciences 050904 information & library sciences business computer |
| Zdroj: | CIKM |
| Popis: | 29th ACM International Conference on Information and Knowledge Management (2020 : Virtual, Online; Ireland) The high cost of constructing test collections led many researchers to develop intelligent document selection methods to find relevant documents with fewer judgments than the standard pooling method requires. In this paper, we conduct a comprehensive set of experiments to evaluate six bandit-based document selection methods, in terms of evaluation reliability, fairness, and reusability of the resultant test collections. In our experiments, the best performing method varies across test collections, showing the importance of using diverse test collections for an accurate performance analysis. Our experiments with six test collections also show that Move-To-Front is the most robust method among the ones we investigate. © 2020 ACM. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|