Construct-Aligned Rating Scales Improve the Reliability of Program Evaluation Data
| Autor: | Linda N. Peterson, Kevin W. Eva, Shayna A. Rusticus |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
Program evaluation
020205 medical informatics media_common.quotation_subject Applied psychology 02 engineering and technology Library and Information Sciences 03 medical and health sciences 0302 clinical medicine Rating scale 0202 electrical engineering electronic engineering information engineering Generalizability theory 030212 general & internal medicine Psychology Sophistication Competence (human resources) media_common |
| Zdroj: | Canadian Journal of Program Evaluation. 33:1-20 |
| ISSN: | 1496-7308 0834-1516 |
| DOI: | 10.3138/cjpe.31094 |
| Popis: | In workplace-based assessment, research has suggested that aligning rating scales with how clinical supervisors naturally conceptualize trainee performance improves reliability and makes assessment more efficient. This study examined the generalizability of those findings by determining if construct alignment improves the reliability of competency-focused program evaluation ratings provided by 135 Year 3 medical students and 185 faculty supervisors. Participants were randomly assigned to complete the Readiness for Clerkship survey using one of two construct-aligned rating scales or a competence-focused rating scale. Mean differences, Spearman correlations, and Generalizability analyses compared the scales’ utility. Relative to the competence-oriented scale, the construct-aligned scales showed greater consistency in the rank order of items between students and faculty. Eight student raters were needed to achieve G=0.8 using the independence scale, 12 were needed using the behavioral/independence scale, and 22 were needed using the competence-oriented scale. Faculty were less consistent in their ratings and required more raters (k=25-32), regardless of the scale used. These results extend previous research into the benefits of construct-aligned scales by suggesting that students offer better ratings when the scales used to collect their judgments are aligned with the constructs of independence and sophistication rather than being phrased in terms of competence. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|