Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation
| Autor: | Peter R. Carroll, Bin Yu, Anobel Y. Odisho, Briton Park, John DeNero, Nicholas Altieri, Matthew R. Cooperberg |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
Urologic Diseases
Pathology medicine.medical_specialty AcademicSubjects/SCI01060 Computer science Health Informatics computer.software_genre Research and Applications Convolutional neural network 03 medical and health sciences 0302 clinical medicine medicine cancer 030212 general & internal medicine information extraction natural language processing Document classification Unstructured data Statistical model prostate cancer Information extraction Data point machine learning 030220 oncology & carcinogenesis pathology AcademicSubjects/SCI01530 F1 score AcademicSubjects/MED00010 Classifier (UML) computer |
| Zdroj: | JAMIA Open JAMIA open, vol 3, iss 3 |
| ISSN: | 2574-2531 |
| Popis: | Objective Cancer is a leading cause of death, but much of the diagnostic information is stored as unstructured data in pathology reports. We aim to improve uncertainty estimates of machine learning-based pathology parsers and evaluate performance in low data settings. Materials and methods Our data comes from the Urologic Outcomes Database at UCSF which includes 3232 annotated prostate cancer pathology reports from 2001 to 2018. We approach 17 separate information extraction tasks, involving a wide range of pathologic features. To handle the diverse range of fields, we required 2 statistical models, a document classification method for pathologic features with a small set of possible values and a token extraction method for pathologic features with a large set of values. For each model, we used isotonic calibration to improve the model’s estimates of its likelihood of being correct. Results Our best document classifier method, a convolutional neural network, achieves a weighted F1 score of 0.97 averaged over 12 fields and our best extraction method achieves an accuracy of 0.93 averaged over 5 fields. The performance saturates as a function of dataset size with as few as 128 data points. Furthermore, while our document classifier methods have reliable uncertainty estimates, our extraction-based methods do not, but after isotonic calibration, expected calibration error drops to below 0.03 for all extraction fields. Conclusions We find that when applying machine learning to pathology parsing, large datasets may not always be needed, and that calibration methods can improve the reliability of uncertainty estimates. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|