Can GPT-3.5 generate and code discharge summaries?

Autor:	Falis M; School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom., Gema AP; School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom., Dong H; Department of Computer Science, University of Exeter, Exeter EX4 4QF, United Kingdom., Daines L; Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh EH16 4UX, United Kingdom., Basetti S; Department of Research, Development and Innovation, National Health Service Highland, Inverness IV2 3JH, United Kingdom., Holder M; Centre for Population Health Sciences, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom., Penfold RS; Ageing and Health, Usher Institute, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom.; Advanced Care Research Centre, The University of Edinburgh, Edinburgh EH16 4UX, United Kingdom., Birch A; School of Informatics, The University of Edinburgh, Edinburgh EH8 9AB, United Kingdom., Alex B; Edinburgh Futures Institute, The University of Edinburgh, Edinburgh EH3 9EF, United Kingdom.; School of Literatures, Languages and Cultures, The University of Edinburgh, Edinburgh EH8 9LH, United Kingdom.
Jazyk:	angličtina
Zdroj:	Journal of the American Medical Informatics Association : JAMIA [J Am Med Inform Assoc] 2024 Oct 01; Vol. 31 (10), pp. 2284-2293.
DOI:	10.1093/jamia/ocae132
Abstrakt:	Objectives: The aim of this study was to investigate GPT-3.5 in generating and coding medical documents with International Classification of Diseases (ICD)-10 codes for data augmentation on low-resource labels. Materials and Methods: Employing GPT-3.5 we generated and coded 9606 discharge summaries based on lists of ICD-10 code descriptions of patients with infrequent (or generation) codes within the MIMIC-IV dataset. Combined with the baseline training set, this formed an augmented training set. Neural coding models were trained on baseline and augmented data and evaluated on an MIMIC-IV test set. We report micro- and macro-F1 scores on the full codeset, generation codes, and their families. Weak Hierarchical Confusion Matrices determined within-family and outside-of-family coding errors in the latter codesets. The coding performance of GPT-3.5 was evaluated on prompt-guided self-generated data and real MIMIC-IV data. Clinicians evaluated the clinical acceptability of the generated documents. Results: Data augmentation results in slightly lower overall model performance but improves performance for the generation candidate codes and their families, including 1 absent from the baseline training data. Augmented models display lower out-of-family error rates. GPT-3.5 identifies ICD-10 codes by their prompted descriptions but underperforms on real data. Evaluators highlight the correctness of generated concepts while suffering in variety, supporting information, and narrative. Discussion and Conclusion: While GPT-3.5 alone given our prompt setting is unsuitable for ICD-10 coding, it supports data augmentation for training neural models. Augmentation positively affects generation code families but mainly benefits codes with existing examples. Augmentation reduces out-of-family errors. Documents generated by GPT-3.5 state prompted concepts correctly but lack variety, and authenticity in narratives. (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)
Databáze:	MEDLINE
Externí odkaz:	Zobrazit plný text záznamu