A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study
BackgroundClinical prediction models suffer from performance drift as the patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively use the old and new data. ObjectiveBased on the paradigm...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2022-11-01
|
| Series: | JMIR Medical Informatics |
| Online Access: | https://medinform.jmir.org/2022/11/e38053 |
| _version_ | 1797734631234076672 |
|---|---|
| author | Xiangzhou Zhang Yunfei Xue Xinyu Su Shaoyong Chen Kang Liu Weiqi Chen Mei Liu Yong Hu |
| author_facet | Xiangzhou Zhang Yunfei Xue Xinyu Su Shaoyong Chen Kang Liu Weiqi Chen Mei Liu Yong Hu |
| author_sort | Xiangzhou Zhang |
| collection | DOAJ |
| description |
BackgroundClinical prediction models suffer from performance drift as the patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively use the old and new data.
ObjectiveBased on the paradigm of transfer learning, we aimed to develop a novel modeling framework that transfers old knowledge to the new environment for prediction tasks, and contributes to performance drift correction.
MethodsThe proposed predictive modeling framework maintains a logistic regression–based stacking ensemble of 2 gradient boosting machine (GBM) models representing old and new knowledge learned from old and new data, respectively (referred to as transfer learning gradient boosting machine [TransferGBM]). The ensemble learning procedure can dynamically balance the old and new knowledge. Using 2010-2017 electronic health record data on a retrospective cohort of 141,696 patients, we validated TransferGBM for hospital-acquired acute kidney injury prediction.
ResultsThe baseline models (ie, transported models) that were trained on 2010 and 2011 data showed significant performance drift in the temporal validation with 2012-2017 data. Refitting these models using updated samples resulted in performance gains in nearly all cases. The proposed TransferGBM model succeeded in achieving uniformly better performance than the refitted models.
ConclusionsUnder the scenario of population shift, incorporating new knowledge while preserving old knowledge is essential for maintaining stable performance. Transfer learning combined with stacking ensemble learning can help achieve a balance of old and new knowledge in a flexible and adaptive way, even in the case of insufficient new data. |
| first_indexed | 2024-03-12T12:47:09Z |
| format | Article |
| id | doaj.art-26c8714824c2499598308be10f584c83 |
| institution | Directory Open Access Journal |
| issn | 2291-9694 |
| language | English |
| last_indexed | 2024-03-12T12:47:09Z |
| publishDate | 2022-11-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Informatics |
| spelling | doaj.art-26c8714824c2499598308be10f584c832023-08-28T23:15:27ZengJMIR PublicationsJMIR Medical Informatics2291-96942022-11-011011e3805310.2196/38053A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort StudyXiangzhou Zhanghttps://orcid.org/0000-0003-3752-0045Yunfei Xuehttps://orcid.org/0000-0001-6871-8228Xinyu Suhttps://orcid.org/0000-0001-7689-1952Shaoyong Chenhttps://orcid.org/0000-0002-9691-9905Kang Liuhttps://orcid.org/0000-0001-9270-1362Weiqi Chenhttps://orcid.org/0000-0001-6550-5024Mei Liuhttps://orcid.org/0000-0002-8036-2110Yong Huhttps://orcid.org/0000-0002-3837-1750 BackgroundClinical prediction models suffer from performance drift as the patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively use the old and new data. ObjectiveBased on the paradigm of transfer learning, we aimed to develop a novel modeling framework that transfers old knowledge to the new environment for prediction tasks, and contributes to performance drift correction. MethodsThe proposed predictive modeling framework maintains a logistic regression–based stacking ensemble of 2 gradient boosting machine (GBM) models representing old and new knowledge learned from old and new data, respectively (referred to as transfer learning gradient boosting machine [TransferGBM]). The ensemble learning procedure can dynamically balance the old and new knowledge. Using 2010-2017 electronic health record data on a retrospective cohort of 141,696 patients, we validated TransferGBM for hospital-acquired acute kidney injury prediction. ResultsThe baseline models (ie, transported models) that were trained on 2010 and 2011 data showed significant performance drift in the temporal validation with 2012-2017 data. Refitting these models using updated samples resulted in performance gains in nearly all cases. The proposed TransferGBM model succeeded in achieving uniformly better performance than the refitted models. ConclusionsUnder the scenario of population shift, incorporating new knowledge while preserving old knowledge is essential for maintaining stable performance. Transfer learning combined with stacking ensemble learning can help achieve a balance of old and new knowledge in a flexible and adaptive way, even in the case of insufficient new data.https://medinform.jmir.org/2022/11/e38053 |
| spellingShingle | Xiangzhou Zhang Yunfei Xue Xinyu Su Shaoyong Chen Kang Liu Weiqi Chen Mei Liu Yong Hu A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study JMIR Medical Informatics |
| title | A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study |
| title_full | A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study |
| title_fullStr | A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study |
| title_full_unstemmed | A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study |
| title_short | A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study |
| title_sort | transfer learning approach to correct the temporal performance drift of clinical prediction models retrospective cohort study |
| url | https://medinform.jmir.org/2022/11/e38053 |
| work_keys_str_mv | AT xiangzhouzhang atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT yunfeixue atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT xinyusu atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT shaoyongchen atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT kangliu atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT weiqichen atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT meiliu atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT yonghu atransferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT xiangzhouzhang transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT yunfeixue transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT xinyusu transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT shaoyongchen transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT kangliu transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT weiqichen transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT meiliu transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy AT yonghu transferlearningapproachtocorrectthetemporalperformancedriftofclinicalpredictionmodelsretrospectivecohortstudy |