Detection of calibration drift in clinical prediction models to inform model updating
| Autor: | Colin G. Walsh, Robert A. Greevy, Michael E. Matheny, Sharon E. Davis, Thomas A. Lasko |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
0303 health sciences
Models Statistical Computer science Calibration curve Calibration (statistics) Health Informatics Predictive analytics computer.software_genre Prognosis Article Computer Science Applications 03 medical and health sciences 0302 clinical medicine Stochastic gradient descent Calibration Use case 030212 general & internal medicine Data mining Focus (optics) Implementation computer Predictive modelling Algorithms 030304 developmental biology |
| Zdroj: | J Biomed Inform |
| ISSN: | 1532-0480 |
| Popis: | Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect