Reliability of a generative artificial intelligence tool for pediatric familial Mediterranean fever: insights from a multicentre expert survey.

Autor: La Bella S; Department of Pediatrics, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy. saveriolabella@outlook.it.; Division of Pediatric Rheumatology, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy. saveriolabella@outlook.it.; Division of Rheumatology and Autoinflammatory Diseases, IRCCS Istituto Giannina Gaslini, Genova, Italy. saveriolabella@outlook.it., Attanasi M; Department of Pediatrics, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy., Porreca A; Laboratory of Biostatistics, Department of Medical, Oral and Biotechnological Sciences, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy., Di Ludovico A; Department of Pediatrics, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy.; Division of Pediatric Rheumatology, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy., Maggio MC; University Department PROMISE 'G. D'Alessandro', University of Palermo, Palermo, Italy., Gallizzi R; Department of Medical of Health Sciences, Magna Graecia University, Catanzaro, Italy., La Torre F; Department of Pediatrics, Giovanni XXIII Pediatric Hospital, University of Bari, Bari, Italy., Rigante D; Department of Life Sciences and Public Health, Fondazione Policlinico Universitario A. Gemelli, Rome and Università Cattolica Sacro Cuore, Rome, Italy., Soscia F; Department of Pediatrics, Sant' Eugenio Hospital, Rome, Italy., Ardenti Morini F; Department of Pediatrics, Sant' Eugenio Hospital, Rome, Italy., Insalaco A; Division of Rheumatology, Bambino Gesù Children's Hospital, Scientific Institute for Research and Health Care, Rome, Italy., Natale MF; Division of Rheumatology, Bambino Gesù Children's Hospital, Scientific Institute for Research and Health Care, Rome, Italy., Chiarelli F; Department of Pediatrics, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy. chiarelli@unich.it., Simonini G; Rheumatology Unit, IRCCS Meyer Children's Hospital, Florence, Italy., De Benedetti F; Division of Rheumatology, Bambino Gesù Children's Hospital, Scientific Institute for Research and Health Care, Rome, Italy., Gattorno M; Division of Rheumatology and Autoinflammatory Diseases, IRCCS Istituto Giannina Gaslini, Genova, Italy., Breda L; Department of Pediatrics, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy.; Division of Pediatric Rheumatology, 'G. D'Annunzio' University of Chieti-Pescara, Chieti, Italy.
Jazyk: angličtina
Zdroj: Pediatric rheumatology online journal [Pediatr Rheumatol Online J] 2024 Aug 23; Vol. 22 (1), pp. 78. Date of Electronic Publication: 2024 Aug 23.
DOI: 10.1186/s12969-024-01011-0
Abstrakt: Background: Artificial intelligence (AI) has become a popular tool for clinical and research use in the medical field. The aim of this study was to evaluate the accuracy and reliability of a generative AI tool on pediatric familial Mediterranean fever (FMF).
Methods: Fifteen questions repeated thrice on pediatric FMF were prompted to the popular generative AI tool Microsoft Copilot with Chat-GPT 4.0. Nine pediatric rheumatology experts rated response accuracy with a blinded mechanism using a Likert-like scale with values from 1 to 5.
Results: Median values for overall responses at the initial assessment ranged from 2.00 to 5.00. During the second assessment, median values spanned from 2.00 to 4.00, while for the third assessment, they ranged from 3.00 to 4.00. Intra-rater variability showed poor to moderate agreement (intraclass correlation coefficient range: -0.151 to 0.534). A diminishing level of agreement among experts over time was documented, as highlighted by Krippendorff's alpha coefficient values, ranging from 0.136 (at the first response) to 0.132 (at the second response) to 0.089 (at the third response). Lastly, experts displayed varying levels of trust in AI pre- and post-survey.
Conclusions: AI has promising implications in pediatric rheumatology, including early diagnosis and management optimization, but challenges persist due to uncertain information reliability and the lack of expert validation. Our survey revealed considerable inaccuracies and incompleteness in AI-generated responses regarding FMF, with poor intra- and extra-rater reliability. Human validation remains crucial in managing AI-generated medical information.
(© 2024. The Author(s).)
Databáze: MEDLINE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje