Reliability of Simulation-Based Assessment for Practicing Physicians: Performance is Context-Specific
| Autor: | Jeffrey B. Cooper, Amanda R. Burden, Samuel DeMaria, Laurence C. Torsher, Matthew S. Shotwell, William R. McIvor, Randolph H. Steadman, Arna Banerjee, John R. Boulet, Elizabeth Sinz, Adam I. Levine, Jason Slagle, Matthew B. Weinger, Christine S. Park, David M. Gaba |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Program evaluation
Generalisability media_common.quotation_subject Population Assessment Continuing medical education Feedback Education Formative assessment Maintenance of Certification 03 medical and health sciences 0302 clinical medicine 030202 anesthesiology Physicians Technical performance measure Humans Computer Simulation education Reliability (statistics) media_common Competency education.field_of_study Teamwork Medical education LC8-6691 Reproducibility of Results 030208 emergency & critical care medicine General Medicine Special aspects of education Anesthesiologists Medicine Clinical Competence Practicing physicians Psychology Simulation Research Article |
| Zdroj: | BMC Medical Education BMC Medical Education, Vol 21, Iss 1, Pp 1-9 (2021) |
| Popis: | Introduction Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. Methods Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. Results The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. Conclusions The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|