| Popis: |
Objective: Little research has been done on the user-centered document ranking approach, especially in a crowdsourcing chart review environment. As the starting point of designing and implementing the next generation of Electronic Medical Record (EMR) search engines, a systematic user study is needed to better understand the users' needs, challenges, and future research directions of EMR search engines. Materials and methods: One primary observation during the user study is the need for a ranking method to better support the so-called ''early stopping'' reviewing strategy (i.e., reviewing only a subset of EMRs of one patient to make the final decision) during the clinical chart reviews. The authors proposed two novel user-centered ranking metrics: ''critical documents'' and ''negative guarantee ratio,'' to better measure the power of a ranking method in supporting the “early stopping” requirements during clinical chart reviews. Results: The evaluation results show that i) traditional information retrieval metrics, such as the precision-at-K, have limitations in guiding the design and development of EMR search engines to better support clinical chart reviews; ii) there is not a global optimal ranking method that fits the needs of different chart reviews and different users; iii) a learning-to-rank approach cannot guarantee a stable and optimal ranking for different chart reviews and different users; and iv) A user-centered ranking metric, such as the negative guarantee ratio (NGR) metric is able to measure the “early-stopping” performance of ranking methods. Conclusions: User-centered ranking metrics can better measure the power of ranking methods in supporting clinical chart reviews. Future research should explore more user-centered ranking metrics and evaluate their impact on real-world EMR search engines. |
