Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models' feasibility in clinical decision-making.

Autor: Sblendorio E; Azienda Ospedaliero-Universitaria Consorziale Policlinico di Bari, Piazza Giulio Cesare, 11, 70124 Bari, Italy; Department of Biomedicine and Prevention, University of Rome 'Tor Vergata', Italy., Dentamaro V; Department of Computer Science, University of Bari 'Aldo Moro', Bari, Italy. Electronic address: https://twitter.com/vincenzoden., Lo Cascio A; Department of Biomedicine and Prevention, University of Rome 'Tor Vergata', Italy; La Maddalena Cancer Center, Via San Lorenzo 312, 90146 Palermo, Italy., Germini F; Department of Biomedicine and Prevention, University of Rome 'Tor Vergata', Italy; Direttore di Distretto Sociosanitario, ASL Bari, Bari, Italy., Piredda M; Department of Medicine and Surgery, Research Unit Nursing Science, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo, 21, 00128 Rome, Italy., Cicolini G; Department of Innovative Technologies in Medicine & Dentistry, 'G.d'Annunzio' University of Chieti - Pescara, Italy.
Jazyk: angličtina
Zdroj: International journal of medical informatics [Int J Med Inform] 2024 Aug; Vol. 188, pp. 105501. Date of Electronic Publication: 2024 May 26.
DOI: 10.1016/j.ijmedinf.2024.105501
Abstrakt: Background: Recent enhancements in Large Language Models (LLMs) such as ChatGPT have exponentially increased user adoption. These models are accessible on mobile devices and support multimodal interactions, including conversations, code generation, and patient image uploads, broadening their utility in providing healthcare professionals with real-time support for clinical decision-making. Nevertheless, many authors have highlighted serious risks that may arise from the adoption of LLMs, principally related to safety and alignment with ethical guidelines.
Objective: To address these challenges, we introduce a novel methodological approach designed to assess the specific feasibility of adopting LLMs within a healthcare area, with a focus on clinical nursing, evaluating their performance and thereby directing their choice. Emphasizing LLMs' adherence to scientific advancements, this approach prioritizes safety and care personalization, according to the "Organization for Economic Co-operation and Development" frameworks for responsible AI. Moreover, its dynamic nature is designed to adapt to future evolutions of LLMs.
Method: Through integrating advanced multidisciplinary knowledge, including Nursing Informatics, and aided by a prospective literature review, seven key domains and specific evaluation items were identified as follows:A Peer Review by experts in Nursing and AI was performed, ensuring scientific rigor and breadth of insights for an essential, reproducible, and coherent methodological approach. By means of a 7-point Likert scale, thresholds are defined in order to classify LLMs as "unusable", "usable with high caution", and "recommended" categories. Nine state of the art LLMs were evaluated using this methodology in clinical oncology nursing decision-making, producing preliminary results. Gemini Advanced, Anthropic Claude 3 and ChatGPT 4 achieved the minimum score of the State of the Art Alignment & Safety domain for classification as "recommended", being also endorsed across all domains. LLAMA 3 70B and ChatGPT 3.5 were classified as "usable with high caution." Others were classified as unusable in this domain.
Conclusion: The identification of a recommended LLM for a specific healthcare area, combined with its critical, prudent, and integrative use, can support healthcare professionals in decision-making processes.
Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
(Copyright © 2024 The Authors. Published by Elsevier B.V. All rights reserved.)
Databáze: MEDLINE
Externí odkaz: