| Popis: |
User simulation is needed for evaluating Interactive Information Retrieval (IIR) Systems. However, for any user simulator to be useful, it must be reliable. In this paper, we propose a novel Tester-based evaluation approach to evaluating the reliability of user simulators, in which we would construct a Tester based on a set of IR systems with an expected performance pattern and apply such a Tester to a user simulator to see if the user simulator would generate the expected performance pattern. We construct multiple Testers and apply them to a set of representative user simulators to empirically study the feasibility and effectiveness of the proposed Tester-based evaluation method. The results show that Tester-based evaluation is a feasible and effective method for evaluating user simulators and selecting reliable ones for evaluating IIR systems. |
