Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study
| Autor: | Louise Puli, Neville Chiavaroli, Sarah Anderson, Leigh Clarke, Michael P Dillon, Emily Ridgewell |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
030506 rehabilitation
Medical education Educational measurement media_common.quotation_subject Rehabilitation Entry Level Australia Rubric Orthodontics Documentation Health Professions (miscellaneous) Agreement 03 medical and health sciences 0302 clinical medicine Competency assessment Humans Portfolio Clinical Competence Educational Measurement Performance indicator 0305 other medical science Psychology 030217 neurology & neurosurgery media_common |
| Zdroj: | Prosthetics & Orthotics International. 45:276-288 |
| ISSN: | 1746-1553 0309-3646 |
| Popis: | BACKGROUND: Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes. OBJECTIVES: To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why. STUDY DESIGN: Mixed methods: explanatory sequential. METHOD: Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement. RESULTS: Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented. CONCLUSIONS: Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|