Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN
Several multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop a...
| المؤلفون الرئيسيون: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| التنسيق: | Journal article |
| اللغة: | English |
| منشور في: |
Nature Publishing Group UK
2024
|
| _version_ | 1826314757675679744 |
|---|---|
| author | Slot, MIE Urquijo Castro, MF Winter - van Rossum, I van Hell, HH Dwyer, D Dazzan, P Maat, A De Haan, L Crespo-Facorro, B Glenthøj, BY Lawrie, SM McDonald, C Gruber, O van Amelsvoort, T Arango, C Kircher, T Nelson, B Galderisi, S Weiser, M Sachs, G Kirschner, M Fleischhacker, WW McGuire, P Koutsouleris, N |
| author_facet | Slot, MIE Urquijo Castro, MF Winter - van Rossum, I van Hell, HH Dwyer, D Dazzan, P Maat, A De Haan, L Crespo-Facorro, B Glenthøj, BY Lawrie, SM McDonald, C Gruber, O van Amelsvoort, T Arango, C Kircher, T Nelson, B Galderisi, S Weiser, M Sachs, G Kirschner, M Fleischhacker, WW McGuire, P Koutsouleris, N |
| author_sort | Slot, MIE |
| collection | OXFORD |
| description | Several multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop and externally validate different models predicting functional outcome after a FEP in the context of a schizophrenia-spectrum disorder (FES), based on a previously published cross-validation and machine learning pipeline. A crossover validation approach was adopted in two large, international cohorts (EUFEST, n = 338, and the PSYSCAN FES cohort, n = 226). Scores on the Global Assessment of Functioning scale (GAF) at 12 month follow-up were dichotomized to differentiate between poor (GAF current < 65) and good outcome (GAF current ≥ 65). Pooled non-linear support vector machine (SVM) classifiers trained on the separate cohorts identified patients with a poor outcome with cross-validated balanced accuracies (BAC) of 65-66%, but BAC dropped substantially when the models were applied to patients from a different FES cohort (BAC = 50–56%). A leave-site-out analysis on the merged sample yielded better performance (BAC = 72%), highlighting the effect of combining data from different study designs to overcome calibration issues and improve model transportability. In conclusion, our results indicate that validation of prediction models in an independent sample is essential in assessing the true value of the model. Future external validation studies, as well as attempts to harmonize data collection across studies, are recommended. |
| first_indexed | 2024-12-09T03:10:37Z |
| format | Journal article |
| id | oxford-uuid:e933f6e5-468b-4adb-afaf-350000480466 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-12-09T03:10:37Z |
| publishDate | 2024 |
| publisher | Nature Publishing Group UK |
| record_format | dspace |
| spelling | oxford-uuid:e933f6e5-468b-4adb-afaf-3500004804662024-10-07T20:09:38ZMultivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCANJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:e933f6e5-468b-4adb-afaf-350000480466EnglishJisc Publications RouterNature Publishing Group UK2024Slot, MIEUrquijo Castro, MFWinter - van Rossum, Ivan Hell, HHDwyer, DDazzan, PMaat, ADe Haan, LCrespo-Facorro, BGlenthøj, BYLawrie, SMMcDonald, CGruber, Ovan Amelsvoort, TArango, CKircher, TNelson, BGalderisi, SWeiser, MSachs, GKirschner, MFleischhacker, WWMcGuire, PKoutsouleris, NSeveral multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop and externally validate different models predicting functional outcome after a FEP in the context of a schizophrenia-spectrum disorder (FES), based on a previously published cross-validation and machine learning pipeline. A crossover validation approach was adopted in two large, international cohorts (EUFEST, n = 338, and the PSYSCAN FES cohort, n = 226). Scores on the Global Assessment of Functioning scale (GAF) at 12 month follow-up were dichotomized to differentiate between poor (GAF current < 65) and good outcome (GAF current ≥ 65). Pooled non-linear support vector machine (SVM) classifiers trained on the separate cohorts identified patients with a poor outcome with cross-validated balanced accuracies (BAC) of 65-66%, but BAC dropped substantially when the models were applied to patients from a different FES cohort (BAC = 50–56%). A leave-site-out analysis on the merged sample yielded better performance (BAC = 72%), highlighting the effect of combining data from different study designs to overcome calibration issues and improve model transportability. In conclusion, our results indicate that validation of prediction models in an independent sample is essential in assessing the true value of the model. Future external validation studies, as well as attempts to harmonize data collection across studies, are recommended. |
| spellingShingle | Slot, MIE Urquijo Castro, MF Winter - van Rossum, I van Hell, HH Dwyer, D Dazzan, P Maat, A De Haan, L Crespo-Facorro, B Glenthøj, BY Lawrie, SM McDonald, C Gruber, O van Amelsvoort, T Arango, C Kircher, T Nelson, B Galderisi, S Weiser, M Sachs, G Kirschner, M Fleischhacker, WW McGuire, P Koutsouleris, N Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title | Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title_full | Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title_fullStr | Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title_full_unstemmed | Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title_short | Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN |
| title_sort | multivariable prediction of functional outcome after first episode psychosis a crossover validation approach in eufest and psyscan |
| work_keys_str_mv | AT slotmie multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT urquijocastromf multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT wintervanrossumi multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT vanhellhh multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT dwyerd multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT dazzanp multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT maata multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT dehaanl multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT crespofacorrob multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT glenthøjby multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT lawriesm multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT mcdonaldc multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT grubero multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT vanamelsvoortt multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT arangoc multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT kirchert multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT nelsonb multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT galderisis multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT weiserm multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT sachsg multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT kirschnerm multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT fleischhackerww multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT mcguirep multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan AT koutsoulerisn multivariablepredictionoffunctionaloutcomeafterfirstepisodepsychosisacrossovervalidationapproachineufestandpsyscan |