Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions.

Autor: Schauber SK; Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway. stefanks@uio.no.; Centre for Educational Measurement (CEMO), Faculty of Educational Sciences, University of Oslo, Oslo, Norway. stefanks@uio.no., Olsen AO; Department of Community Medicine and Global Health, Institute of Health and Society, University of Oslo, Oslo, Norway., Werner EL; Department of General Practice, Institute of Health and Society, University of Oslo, Oslo, Norway., Magelssen M; Centre for Medical Ethics, Institute of Health and Society, University of Oslo, Oslo, Norway.
Jazyk: angličtina
Zdroj: Advances in health sciences education : theory and practice [Adv Health Sci Educ Theory Pract] 2024 Nov; Vol. 29 (5), pp. 1749-1767. Date of Electronic Publication: 2024 Apr 23.
DOI: 10.1007/s10459-024-10328-0
Abstrakt: Introduction: Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts.
Methods: We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics.
Results: We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs.
Discussion: Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices.
(© 2024. The Author(s).)
Databáze: MEDLINE
Externí odkaz: