Comparison of Natural Language Processing Rules-Based and Machine-Learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain
Autor: | Nancy Organ, Patrick J. Heagerty, W. Katherine Tan, Brent Griffith, Hannu Huhdanpaa, Kathryn T. James, David R. Nerenz, David Carrell, Saeed Hassanpour, Pradeep Suri, Sean D. Rundell, Eric N. Meier, David F. Kallmes, Karen J. Sherman, Curtis P. Langlotz, Jeffrey G. Jarvik, Patrick H. Luetmer |
---|---|
Jazyk: | angličtina |
Rok vydání: | 2018 |
Předmět: |
Adult
Male Radiography Lumbar vertebrae computer.software_genre Sensitivity and Specificity Article Cohort Studies Machine Learning 03 medical and health sciences 0302 clinical medicine medicine Humans Radiology Nuclear Medicine and imaging 030212 general & internal medicine Aged Natural Language Processing Aged 80 and over Lumbar Vertebrae Receiver operating characteristic business.industry Middle Aged Low back pain Magnetic Resonance Imaging Data set medicine.anatomical_structure Lumbar spine Female Artificial intelligence medicine.symptom business computer Low Back Pain 030217 neurology & neurosurgery Natural language processing Cohort study Healthcare system |
Popis: | Rationale and Objectives To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems. Materials and Methods We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N = 178,333 reports, we randomly selected N = 871 to form a reference-standard dataset, consisting of N = 413 x-ray reports and N = 458 MR reports. Using standardized criteria, four spine experts annotated the presence of 26 findings, where 71 reports were annotated by all four experts and 800 were each annotated by two experts. We calculated inter-rater agreement and finding prevalence from annotated data. We randomly split the annotated data into development (80%) and testing (20%) sets. We developed an NLP system from both rule-based and machine-learned models. We validated the system using accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). Results The multirater annotated dataset achieved inter-rater agreement of Cohen's kappa > 0.60 (substantial agreement) for 25 of 26 findings, with finding prevalence ranging from 3% to 89%. In the testing sample, rule-based and machine-learned predictions both had comparable average specificity (0.97 and 0.95, respectively). The machine-learned approach had a higher average sensitivity (0.94, compared to 0.83 for rules-based), and a higher overall AUC (0.98, compared to 0.90 for rules-based). Conclusions Our NLP system performed well in identifying the 26 lumbar spine findings, as benchmarked by reference-standard annotation by medical experts. Machine-learned models provided substantial gains in model sensitivity with slight loss of specificity, and overall higher AUC. |
Databáze: | OpenAIRE |
Externí odkaz: |