A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population.

Autor: Urquhart E; School of Computer Science, University of Galway, Galway, Ireland., Ryan J; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Hartigan S; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Nita C; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Hanley C; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Moran P; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Bates J; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Jooste R; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland., Judge C; School of Medicine, University of Galway, Galway, Ireland., Laffey JG; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland.; Anaesthesia and Intensive Care Medicine, School of Medicine, University of Galway, Galway, Ireland., Madden MG; School of Computer Science, University of Galway, Galway, Ireland., McNicholas BA; Department of Anaesthesia and Intensive Care Medicine, University Hospital Galway, Galway, H91 YR71, Ireland. bmcnicholas@universityofgalway.ie.; Anaesthesia and Intensive Care Medicine, School of Medicine, University of Galway, Galway, Ireland. bmcnicholas@universityofgalway.ie.
Jazyk: angličtina
Zdroj: Intensive care medicine experimental [Intensive Care Med Exp] 2024 Aug 16; Vol. 12 (1), pp. 71. Date of Electronic Publication: 2024 Aug 16.
DOI: 10.1186/s40635-024-00656-1
Abstrakt: Background: Artificial intelligence, through improved data management and automated summarisation, has the potential to enhance intensive care unit (ICU) care. Large language models (LLMs) can interrogate and summarise large volumes of medical notes to create succinct discharge summaries. In this study, we aim to investigate the potential of LLMs to accurately and concisely synthesise ICU discharge summaries.
Methods: Anonymised clinical notes from ICU admissions were used to train and validate a prompting structure in three separate LLMs (ChatGPT, GPT-4 API and Llama 2) to generate concise clinical summaries. Summaries were adjudicated by staff intensivists on ability to identify and appropriately order a pre-defined list of important clinical events as well as readability, organisation, succinctness, and overall rank.
Results: In the development phase, text from five ICU episodes was used to develop a series of prompts to best capture clinical summaries. In the testing phase, a summary produced by each LLM from an additional six ICU episodes was utilised for evaluation. Overall ability to identify a pre-defined list of important clinical events in the summary was 41.5 ± 15.2% for GPT-4 API, 19.2 ± 20.9% for ChatGPT and 16.5 ± 14.1% for Llama2 (p = 0.002). GPT-4 API followed by ChatGPT had the highest score to appropriately order a pre-defined list of important clinical events in the summary as well as readability, organisation, succinctness, and overall rank, whilst Llama2 scored lowest for all. GPT-4 API produced minor hallucinations, which were not present in the other models.
Conclusion: Differences exist in large language model performance in readability, organisation, succinctness, and sequencing of clinical events compared to others. All encountered issues with narrative coherence and omitted key clinical data and only moderately captured all clinically meaningful data in the correct order. However, these technologies suggest future potential for creating succinct discharge summaries.
(© 2024. The Author(s).)
Databáze: MEDLINE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje