Natural Language Processing and Assessment of Resident Feedback Quality
| Autor: | Quintin P. Solano, Laura Hayward, Erkin Otles, Kenneth L. Abbott, Marcus Kunzmann, Samantha L. Ahle, Brian C. George, Zoey Chopra, Daniel E. Kendrick, Mary C. Schuller, Kathryn Quanstrom |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Formative Feedback
Computer science media_common.quotation_subject Logistic regression computer.software_genre Feedback Education Academic institution 03 medical and health sciences 0302 clinical medicine Professional learning community Humans Quality (business) 030212 general & internal medicine Natural Language Processing media_common Performance feedback Receiver operating characteristic business.industry Internship and Residency Residency program Mobile Applications Confidence interval 030220 oncology & carcinogenesis Surgery Artificial intelligence business computer Natural language processing |
| Zdroj: | Journal of Surgical Education. 78:e72-e77 |
| ISSN: | 1931-7204 |
| Popis: | OBJECTIVE To validate the performance of a natural language processing (NLP) model in characterizing the quality of feedback provided to surgical trainees. DESIGN Narrative surgical resident feedback transcripts were collected from a large academic institution and classified for quality by trained coders. 75% of classified transcripts were used to train a logistic regression NLP model and 25% were used for testing the model. The NLP model was trained by uploading classified transcripts and tested using unclassified transcripts. The model then classified those transcripts into dichotomized high- and low- quality ratings. Model performance was primarily assessed in terms of accuracy and secondary performance measures including sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). SETTING A surgical residency program based in a large academic medical center. PARTICIPANTS All surgical residents who received feedback via the Society for Improving Medical Professional Learning smartphone application (SIMPL, Boston, MA) in August 2019. RESULTS The model classified the quality (high vs. low) of 2,416 narrative feedback transcripts with an accuracy of 0.83 (95% confidence interval: 0.80, 0.86), sensitivity of 0.37 (0.33, 0.45), specificity of 0.97 (0.96, 0.98), and an area under the receiver operating characteristic curve of 0.86 (0.83, 0.87). CONCLUSIONS The NLP model classified the quality of operative performance feedback with high accuracy and specificity. NLP offers residency programs the opportunity to efficiently measure feedback quality. This information can be used for feedback improvement efforts and ultimately, the education of surgical trainees. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect