Using Automated Scoring to Evaluate Written Responses in English and French on a High-Stakes Clinical Competency Examination
| Autor: | Syed Latifi, Mark J. Gierl, André-Philippe Boulais, André De Champlain |
|---|---|
| Rok vydání: | 2015 |
| Předmět: |
Educational measurement
Canada 020205 medical informatics Clinical Decision-Making 02 engineering and technology Automated essay scoring computer.software_genre Clinical decision making 0202 electrical engineering electronic engineering information engineering Medicine Humans Reliability (statistics) Electronic Data Processing business.industry Health Policy 05 social sciences 050301 education Reproducibility of Results Translating Licensure Medical Competency assessment Artificial intelligence Data mining Clinical Competence Educational Measurement Clinical competence business 0503 education computer Natural language processing |
| Zdroj: | Evaluationthe health professions. 39(1) |
| ISSN: | 1552-3918 |
| Popis: | We present a framework for technology-enhanced scoring of bilingual clinical decision-making (CDM) questions using an open-source scoring technology and evaluate the strength of the proposed framework using operational data from the Medical Council of Canada Qualifying Examination. Candidates’ responses from six write-in CDM questions were used to develop a three-stage–automated scoring framework. In Stage 1, the linguistic features from CDM responses were extracted. In Stage 2, supervised machine learning techniques were employed for developing the scoring models. In Stage 3, responses to six English and French CDM questions were scored using the scoring models from Stage 2. Of the 8,007 English and French CDM responses, 7,643 were accurately scored with an agreement rate of 95.4% between human and computer scoring. This result serves as an improvement of 5.4% when compared with the human inter-rater reliability. Our framework yielded scores similar to those of expert physician markers and could be used for clinical competency assessment. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|