RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
| Autor: | Joël Kuiper, Byron C. Wallace, Iain J. Marshall |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2016 |
| Předmět: |
bias
Computer science randomized controlled trials as topic Health Informatics Bias assessment computer.software_genre Research and Applications Machine Learning 03 medical and health sciences 0302 clinical medicine Text mining systematic review Relevance (information retrieval) 030212 general & internal medicine natural language processing Clinical Trials as Topic Information retrieval business.industry Workload data mining Clinical trial Review Literature as Topic Systematic review Databases as Topic Artificial intelligence business computer 030217 neurology & neurosurgery Natural language processing Algorithms |
| Zdroj: | Journal of the American Medical Informatics Association, 23(1), 193-201. B M J PUBLISHING GROUP Journal of the American Medical Informatics Association : JAMIA Marshall, I, Kuiper, J & Wallace, B C 2016, ' RobotReviewer : Evaluation of a System for Automatically Assessing Bias in Clinical Trials ', Journal of the American Medical Informatics Association : JAMIA, vol. 23, no. 1, pp. 193-201 . https://doi.org/10.1093/jamia/ocv044 |
| ISSN: | 1067-5027 |
| Popis: | Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments.Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR.Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was v 78.3% with CDSR).Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|