Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression
Abstract Background The debate of whether machine learning models offer advantages over standard statistical methods when making predictions is ongoing. We discuss the use of a meta-learner model combining both approaches as an alternative. Methods To illustrate the development of a meta-learner, we...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-05-01
|
| Series: | BMC Psychiatry |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12888-022-03986-0 |
| _version_ | 1828807358488248320 |
|---|---|
| author | Qiang Liu Georgia Salanti Franco De Crescenzo Edoardo Giuseppe Ostinelli Zhenpeng Li Anneka Tomlinson Andrea Cipriani Orestis Efthimiou |
| author_facet | Qiang Liu Georgia Salanti Franco De Crescenzo Edoardo Giuseppe Ostinelli Zhenpeng Li Anneka Tomlinson Andrea Cipriani Orestis Efthimiou |
| author_sort | Qiang Liu |
| collection | DOAJ |
| description | Abstract Background The debate of whether machine learning models offer advantages over standard statistical methods when making predictions is ongoing. We discuss the use of a meta-learner model combining both approaches as an alternative. Methods To illustrate the development of a meta-learner, we used a dataset of 187,757 people with depression. Using 31 variables, we aimed to predict two outcomes measured 60 days after initiation of antidepressant treatment: severity of depressive symptoms (continuous) and all-cause dropouts (binary). We fitted a ridge regression and a multi-layer perceptron (MLP) deep neural network as two separate prediction models (“base-learners”). We then developed two “meta-learners”, combining predictions from the two base-learners. To compare the performance across the different methods, we calculated mean absolute error (MAE, for continuous outcome) and the area under the receiver operating characteristic curve (AUC, for binary outcome) using bootstrapping. Results Compared to the best performing base-learner (MLP base-learner, MAE at 4.63, AUC at 0.59), the best performing meta-learner showed a 2.49% decrease in MAE at 4.52 for the continuous outcome and a 6.47% increase in AUC at 0.60 for the binary outcome. Conclusions A meta-learner approach may effectively combine multiple prediction models. Choosing between statistical and machine learning models may not be necessary in practice. |
| first_indexed | 2024-12-12T08:24:29Z |
| format | Article |
| id | doaj.art-02efff6e520642fbb92a6be66ea97c22 |
| institution | Directory Open Access Journal |
| issn | 1471-244X |
| language | English |
| last_indexed | 2024-12-12T08:24:29Z |
| publishDate | 2022-05-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Psychiatry |
| spelling | doaj.art-02efff6e520642fbb92a6be66ea97c222022-12-22T00:31:18ZengBMCBMC Psychiatry1471-244X2022-05-0122111010.1186/s12888-022-03986-0Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depressionQiang Liu0Georgia Salanti1Franco De Crescenzo2Edoardo Giuseppe Ostinelli3Zhenpeng Li4Anneka Tomlinson5Andrea Cipriani6Orestis Efthimiou7Department of Psychiatry, Warneford Hospital, University of OxfordInstitute of Social and Preventive Medicine, University of BernDepartment of Psychiatry, Warneford Hospital, University of OxfordDepartment of Psychiatry, Warneford Hospital, University of OxfordDepartment of Psychiatry, Warneford Hospital, University of OxfordDepartment of Psychiatry, Warneford Hospital, University of OxfordDepartment of Psychiatry, Warneford Hospital, University of OxfordDepartment of Psychiatry, Warneford Hospital, University of OxfordAbstract Background The debate of whether machine learning models offer advantages over standard statistical methods when making predictions is ongoing. We discuss the use of a meta-learner model combining both approaches as an alternative. Methods To illustrate the development of a meta-learner, we used a dataset of 187,757 people with depression. Using 31 variables, we aimed to predict two outcomes measured 60 days after initiation of antidepressant treatment: severity of depressive symptoms (continuous) and all-cause dropouts (binary). We fitted a ridge regression and a multi-layer perceptron (MLP) deep neural network as two separate prediction models (“base-learners”). We then developed two “meta-learners”, combining predictions from the two base-learners. To compare the performance across the different methods, we calculated mean absolute error (MAE, for continuous outcome) and the area under the receiver operating characteristic curve (AUC, for binary outcome) using bootstrapping. Results Compared to the best performing base-learner (MLP base-learner, MAE at 4.63, AUC at 0.59), the best performing meta-learner showed a 2.49% decrease in MAE at 4.52 for the continuous outcome and a 6.47% increase in AUC at 0.60 for the binary outcome. Conclusions A meta-learner approach may effectively combine multiple prediction models. Choosing between statistical and machine learning models may not be necessary in practice.https://doi.org/10.1186/s12888-022-03986-0DepressionPHQ-9DropoutMachine learningStatistical model |
| spellingShingle | Qiang Liu Georgia Salanti Franco De Crescenzo Edoardo Giuseppe Ostinelli Zhenpeng Li Anneka Tomlinson Andrea Cipriani Orestis Efthimiou Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression BMC Psychiatry Depression PHQ-9 Dropout Machine learning Statistical model |
| title | Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression |
| title_full | Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression |
| title_fullStr | Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression |
| title_full_unstemmed | Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression |
| title_short | Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression |
| title_sort | development and validation of a meta learner for combining statistical and machine learning prediction models in individuals with depression |
| topic | Depression PHQ-9 Dropout Machine learning Statistical model |
| url | https://doi.org/10.1186/s12888-022-03986-0 |
| work_keys_str_mv | AT qiangliu developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT georgiasalanti developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT francodecrescenzo developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT edoardogiuseppeostinelli developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT zhenpengli developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT annekatomlinson developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT andreacipriani developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression AT orestisefthimiou developmentandvalidationofametalearnerforcombiningstatisticalandmachinelearningpredictionmodelsinindividualswithdepression |