Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making.

Autor: Civettini I; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Zappaterra A; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy.; Department of Haematology and Bone Marrow Transplantation Unit, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy., Granelli BM; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Rindone G; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Aroldi A; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Bonfanti S; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Colombo F; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Fedele M; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Grillo G; Department of Haematology and Bone Marrow Transplantation Unit, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy., Parma M; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Perfetti P; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Terruzzi E; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Gambacorti-Passerini C; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy.; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy., Ramazzotti D; Department of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy., Cavalca F; Department of Haematology and Bone Marrow Trasplantation Unit, Fondazione IRCCS San Gerardo dei Tintori, Monza, Italy.
Jazyk: angličtina
Zdroj: British journal of haematology [Br J Haematol] 2024 Apr; Vol. 204 (4), pp. 1523-1528. Date of Electronic Publication: 2023 Dec 09.
DOI: 10.1111/bjh.19200
Abstrakt: In a first-of-its-kind study, we assessed the capabilities of large language models (LLMs) in making complex decisions in haematopoietic stem cell transplantation. The evaluation was conducted not only for Generative Pre-trained Transformer 4 (GPT-4) but also conducted on other artificial intelligence models: PaLm 2 and Llama-2. Using detailed haematological histories that include both clinical, molecular and donor data, we conducted a triple-blind survey to compare LLMs to haematology residents. We found that residents significantly outperformed LLMs (p = 0.02), particularly in transplant eligibility assessment (p = 0.01). Our triple-blind methodology aimed to mitigate potential biases in evaluating LLMs and revealed both their promise and limitations in deciphering complex haematological clinical scenarios.
(© 2023 The Authors. British Journal of Haematology published by British Society for Haematology and John Wiley & Sons Ltd.)
Databáze: MEDLINE
Externí odkaz: