Assessing ChatGPT's theoretical knowledge and prescriptive accuracy in bacterial infections: a comparative study with infectious diseases residents and specialists.

Autor: De Vito A; Unit of Infectious Diseases, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy. andreadevitoaho@gmail.com.; PhD School in Biomedical Science, Biomedical Science Department, University of Sassari, Sassari, Italy. andreadevitoaho@gmail.com., Geremia N; Unit of Infectious Diseases, Department of Clinical Medicine, Ospedale dell'Angelo, Venice, Italy.; Unit of Infectious Diseases, Department of Clinical Medicine, Ospedale Civile S.S. Giovanni e Paolo, Venice, Italy., Marino A; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, ARNAS Garibaldi Hospital, University of Catania, Catania, Italy., Bavaro DF; Infectious Diseases Unit - IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, Milan, 20089, Italy.; Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, Milan, 20090, Italy., Caruana G; Infectious Diseases Service, Cantonal Hospital of Sion and Institut Central des Hôpitaux (ICH), Sion, Switzerland.; Institute of Microbiology, Department of Laboratory Medicine and Pathology, Lausanne University Hospital, Lausanne, Switzerland., Meschiari M; University of Modena and Reggio Emilia, Modena, Italy., Colpani A; Unit of Infectious Diseases, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy., Mazzitelli M; Infectious and Tropical Diseases Unit, Padua University Hospital, Padua, Italy., Scaglione V; Infectious and Tropical Diseases Unit, Padua University Hospital, Padua, Italy., Venanzi Rullo E; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy., Fiore V; Unit of Infectious Diseases, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy., Fois M; Unit of Infectious Diseases, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy., Campanella E; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, ARNAS Garibaldi Hospital, University of Catania, Catania, Italy.; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy., Pistarà E; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, ARNAS Garibaldi Hospital, University of Catania, Catania, Italy.; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy., Faltoni M; University of Modena and Reggio Emilia, Modena, Italy., Nunnari G; Unit of Infectious Diseases, Department of Clinical and Experimental Medicine, ARNAS Garibaldi Hospital, University of Catania, Catania, Italy., Cattelan A; Infectious and Tropical Diseases Unit, Padua University Hospital, Padua, Italy., Mussini C; University of Modena and Reggio Emilia, Modena, Italy., Bartoletti M; Infectious Diseases Unit - IRCCS Humanitas Research Hospital, Via Manzoni 56, Rozzano, Milan, 20089, Italy.; Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, Pieve Emanuele, Milan, 20090, Italy., Vaira LA; Maxillofacial Surgery Unit, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy., Madeddu G; Unit of Infectious Diseases, Department of Medicine, Surgery, and Pharmacy, University of Sassari, Sassari, Italy.
Jazyk: angličtina
Zdroj: Infection [Infection] 2024 Jul 12. Date of Electronic Publication: 2024 Jul 12.
DOI: 10.1007/s15010-024-02350-6
Abstrakt: Objectives: Advancements in Artificial Intelligence(AI) have made platforms like ChatGPT increasingly relevant in medicine. This study assesses ChatGPT's utility in addressing bacterial infection-related questions and antibiogram-based clinical cases.
Methods: This study involved a collaborative effort involving infectious disease (ID) specialists and residents. A group of experts formulated six true/false, six open-ended questions, and six clinical cases with antibiograms for four types of infections (endocarditis, pneumonia, intra-abdominal infections, and bloodstream infection) for a total of 96 questions. The questions were submitted to four senior residents and four specialists in ID and inputted into ChatGPT-4 and a trained version of ChatGPT-4. A total of 720 responses were obtained and reviewed by a blinded panel of experts in antibiotic treatments. They evaluated the responses for accuracy and completeness, the ability to identify correct resistance mechanisms from antibiograms, and the appropriateness of antibiotics prescriptions.
Results: No significant difference was noted among the four groups for true/false questions, with approximately 70% correct answers. The trained ChatGPT-4 and ChatGPT-4 offered more accurate and complete answers to the open-ended questions than both the residents and specialists. Regarding the clinical case, we observed a lower accuracy from ChatGPT-4 to recognize the correct resistance mechanism. ChatGPT-4 tended not to prescribe newer antibiotics like cefiderocol or imipenem/cilastatin/relebactam, favoring less recommended options like colistin. Both trained- ChatGPT-4 and ChatGPT-4 recommended longer than necessary treatment periods (p-value = 0.022).
Conclusions: This study highlights ChatGPT's capabilities and limitations in medical decision-making, specifically regarding bacterial infections and antibiogram analysis. While ChatGPT demonstrated proficiency in answering theoretical questions, it did not consistently align with expert decisions in clinical case management. Despite these limitations, the potential of ChatGPT as a supportive tool in ID education and preliminary analysis is evident. However, it should not replace expert consultation, especially in complex clinical decision-making.
(© 2024. The Author(s).)
Databáze: MEDLINE
Externí odkaz: