Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery.

Autor: Wissel BD; Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Greiner HM; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Neurology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Glauser TA; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Neurology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Holland-Bouley KD; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Neurology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Mangano FT; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Neurosurgery, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Santel D; Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Faist R; Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Zhang N; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Biostatistics & Epidemiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Pestian JP; Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio., Szczesniak RD; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Biostatistics & Epidemiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio., Dexheimer JW; Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.; Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.; Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
Jazyk: angličtina
Zdroj: Epilepsia [Epilepsia] 2020 Jan; Vol. 61 (1), pp. 39-48. Date of Electronic Publication: 2019 Nov 29.
DOI: 10.1111/epi.16398
Abstrakt: Objective: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores.
Methods: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One- to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review.
Results: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6.
Significance: An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting.
(Wiley Periodicals, Inc. © 2019 International League Against Epilepsy.)
Databáze: MEDLINE
Externí odkaz: