Machine learning clinical prediction models for acute kidney injury: the impact of baseline creatinine on prediction efficacy.

Autor: Kamel Rahimi A; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia. amir.kamel@uq.edu.au.; Digital Health Cooperative Research Centre, Australian Government, Sydney, NSW, Australia. amir.kamel@uq.edu.au., Ghadimi M; The School of Mathematics and Physics, The University of Queensland, St Lucia, Brisbane, 4072, Australia., van der Vegt AH; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia., Canfell OJ; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Digital Health Cooperative Research Centre, Australian Government, Sydney, NSW, Australia.; UQ Business School, The University of Queensland, St Lucia, Brisbane, 4072, Australia., Pole JD; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Dalla Lana School of Public Health, The University of Toronto, Toronto, Canada.; ICES, Toronto, Canada., Sullivan C; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Metro North Hospital and Health Service, Department of Health, Queensland Government, Herston, Brisbane, 4006, Australia., Shrapnel S; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; The School of Mathematics and Physics, The University of Queensland, St Lucia, Brisbane, 4072, Australia.
Jazyk: angličtina
Zdroj: BMC medical informatics and decision making [BMC Med Inform Decis Mak] 2023 Oct 09; Vol. 23 (1), pp. 207. Date of Electronic Publication: 2023 Oct 09.
DOI: 10.1186/s12911-023-02306-0
Abstrakt: Background: There are many Machine Learning (ML) models which predict acute kidney injury (AKI) for hospitalised patients. While a primary goal of these models is to support clinical decision-making, the adoption of inconsistent methods of estimating baseline serum creatinine (sCr) may result in a poor understanding of these models' effectiveness in clinical practice. Until now, the performance of such models with different baselines has not been compared on a single dataset. Additionally, AKI prediction models are known to have a high rate of false positive (FP) events regardless of baseline methods. This warrants further exploration of FP events to provide insight into potential underlying reasons.
Objective: The first aim of this study was to assess the variance in performance of ML models using three methods of baseline sCr on a retrospective dataset. The second aim was to conduct an error analysis to gain insight into the underlying factors contributing to FP events.
Materials and Methods: The Intensive Care Unit (ICU) patients of the Medical Information Mart for Intensive Care (MIMIC)-IV dataset was used with the KDIGO (Kidney Disease Improving Global Outcome) definition to identify AKI episodes. Three different methods of estimating baseline sCr were defined as (1) the minimum sCr, (2) the Modification of Diet in Renal Disease (MDRD) equation and the minimum sCr and (3) the MDRD equation and the mean of preadmission sCr. For the first aim of this study, a suite of ML models was developed for each baseline and the performance of the models was assessed. An analysis of variance was performed to assess the significant difference between eXtreme Gradient Boosting (XGB) models across all baselines. To address the second aim, Explainable AI (XAI) methods were used to analyse the XGB errors with Baseline 3.
Results: Regarding the first aim, we observed variances in discriminative metrics and calibration errors of ML models when different baseline methods were adopted. Using Baseline 1 resulted in a 14% reduction in the f1 score for both Baseline 2 and Baseline 3. There was no significant difference observed in the results between Baseline 2 and Baseline 3. For the second aim, the FP cohort was analysed using the XAI methods which led to relabelling data with the mean of sCr in 180 to 0 days pre-ICU as the preferred sCr baseline method. The XGB model using this relabelled data achieved an AUC of 0.85, recall of 0.63, precision of 0.54 and f1 score of 0.58. The cohort size was 31,586 admissions, of which 5,473 (17.32%) had AKI.
Conclusion: In the absence of a widely accepted method of baseline sCr, AKI prediction studies need to consider the impact of different baseline methods on the effectiveness of ML models and their potential implications in real-world implementations. The utilisation of XAI methods can be effective in providing insight into the occurrence of prediction errors. This can potentially augment the success rate of ML implementation in routine care.
(© 2023. BioMed Central Ltd., part of Springer Nature.)
Databáze: MEDLINE
Externí odkaz:
Nepřihlášeným uživatelům se plný text nezobrazuje