Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery
| Autor: | Grace L. Paley, Rebecca Grove, Steven M. Shields, Tony N. Pira, Mae O. Gordon, Michael V. Stock, Jack Pruett, Evan L. Waxman, Susan M. Culican, Tejas C. Sekhar, Bradley Wilson |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Washington
medicine.medical_specialty Longitudinal study medicine.medical_treatment Cataract Extraction Crowdsourcing Article Cataract Education surgical competence 03 medical and health sciences 0302 clinical medicine medicine Humans 030212 general & internal medicine Longitudinal Studies Reliability (statistics) business.industry Practice-Based Learning and Improvement Training level Construct validity Internship and Residency Reproducibility of Results cataract surgery Cataract surgery Test (assessment) Ophthalmology Inter-rater reliability surgical assessment Cross-Sectional Studies 030220 oncology & carcinogenesis phacoemulsification Systems-Based Practice Physical therapy Surgery Patient Care Clinical Competence business Medical Knowledge |
| Zdroj: | Journal of surgical education |
| ISSN: | 1878-7452 |
| Popis: | OBJECTIVE: To test whether crowdsourced lay raters can accurately assess cataract surgical skills. DESIGN: Two-armed study: independent cross-sectional and longitudinal cohorts. SETTING: Washington University Department of Ophthalmology. PARTICIPANTS AND METHODS: Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study: 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study: 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores. RESULTS: Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = −0.855, p < 0.0001) and expert score (r = −0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r2 = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r2 = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd. CONCLUSIONS: Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect