Correcting for Rater Effects in Operating Room Surgical Skills Assessment.

Autor: Chou R; Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A., Naz H; Dugoni School of Dentistry, University of Pacific, San Francisco, California, U.S.A., Boahene KDO; Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.; Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A., Maxwell JH; Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A.; ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A., Wanamaker JR; Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A.; ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A., Byrne PJ; Head and Neck Institute, Cleveland Clinic, Cleveland, Ohio, U.S.A., Papel ID; Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.; Aesthetic Center at Woodholme, Baltimore, Maryland, U.S.A., Kontis TC; Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.; Aesthetic Center at Woodholme, Baltimore, Maryland, U.S.A., Hager GD; Malone Center for Engineering in Healthcare, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A.; Department of Computer Science, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A., Ishii LE; Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.; Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A., Malekzadeh S; Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A.; ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A., Vedula SS; Malone Center for Engineering in Healthcare, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A., Ishii M; Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.
Jazyk: angličtina
Zdroj: The Laryngoscope [Laryngoscope] 2024 Aug; Vol. 134 (8), pp. 3548-3554. Date of Electronic Publication: 2024 Mar 12.
DOI: 10.1002/lary.31391
Abstrakt: Objective: To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.
Methods: We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.
Results: Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.
Conclusion: Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments.
Level of Evidence: N/A Laryngoscope, 134:3548-3554, 2024.
(© 2024 The American Laryngological, Rhinological and Otological Society, Inc.)
Databáze: MEDLINE
Externí odkaz: