Exploring how feedback reflects entrustment decisions using artificial intelligence.

Autor: Gin BC; Department of Pediatrics, University of California San Francisco, San Francisco, CA, USA., Ten Cate O; Utrecht Center for Research and Development of Health Professions Education, University Medical Center, Utrecht, The Netherlands.; Department of Medicine, University of California San Francisco, San Francisco, CA, USA., O'Sullivan PS; Department of Medicine, University of California San Francisco, San Francisco, CA, USA.; Department of Surgery, University of California San Francisco, San Francisco, CA, USA., Hauer KE; Department of Medicine, University of California San Francisco, San Francisco, CA, USA., Boscardin C; Department of Medicine, University of California San Francisco, San Francisco, CA, USA.; Department of Anesthesia, University of California San Francisco, San Francisco, CA, USA.
Jazyk: angličtina
Zdroj: Medical education [Med Educ] 2022 Mar; Vol. 56 (3), pp. 303-311. Date of Electronic Publication: 2021 Dec 01.
DOI: 10.1111/medu.14696
Abstrakt: Context: Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs.
Methods: In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment-based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction.
Results: We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently.
Conclusions: Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal-setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.
(© 2021 Association for the Study of Medical Education and John Wiley & Sons Ltd.)
Databáze: MEDLINE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje