| Popis: |
Traditional batch evaluation metrics assume that user interaction\ud with search results is limited to scanning down a ranked list. However,\ud modern search interfaces come with additional elements supporting\ud result list refinement (RLR) through facets and filters, making\ud user search behavior increasingly dynamic. We develop an\ud evaluation framework that takes a step beyond the interaction assumption\ud of traditional evaluation metrics and allows for batch evaluation\ud of systems with and without RLR elements. In our framework\ud we model user interaction as switching between different sublists.\ud This provides a measure of user effort based on the joint effect\ud of user interaction with RLR elements and result quality.\ud We validate our framework by conducting a user study and comparing\ud model predictions with real user performance. Our model\ud predictions show significant positive correlation with real user effort.\ud Further, in contrast to traditional evaluation metrics, the predictions\ud using our framework, of when users stand to benefit from\ud RLR elements, reflect findings from our user study.\ud Finally, we use the framework to investigate under what conditions\ud systems with and without RLR elements are likely to be effective.\ud We simulate varying conditions concerning ranking quality,users, task and interface properties demonstrating a cost-effectiveway to study whole system performance. |
