Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback
| Autor: | Nina W. Zhao, Bridget C. O’Brien, Lindsey M. Haddock |
|---|---|
| Rok vydání: | 2022 |
| Předmět: |
Evidence-based practice
Response process Interpretation (philosophy) media_common.quotation_subject Applied psychology Internship and Residency Cognition Procedural memory Feedback Education Variety (cybernetics) Formative assessment Humans Surgery Clinical Competence Workplace Psychology Autonomy media_common |
| Zdroj: | Journal of Surgical Education. 79:475-484 |
| ISSN: | 1931-7204 |
| DOI: | 10.1016/j.jsurg.2021.09.007 |
| Popis: | OBJECTIVE Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery. DESIGN Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes. SETTING Single tertiary academic medical center. PARTICIPANTS Eight residents and 13 faculty within the Department of Otolaryngology—Head and Neck Surgery participating in a 6-month pilot of SIMPL. RESULTS We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty. CONCLUSIONS Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect