Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics
Abstract Background Many studies in radiomics are using feature selection methods to identify the most predictive features. At the same time, they employ cross-validation to estimate the performance of the developed models. However, if the feature selection is performed before the cross-validation,...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2021-11-01
|
| Series: | Insights into Imaging |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13244-021-01115-1 |
| _version_ | 1818343039734644736 |
|---|---|
| author | Aydin Demircioğlu |
| author_facet | Aydin Demircioğlu |
| author_sort | Aydin Demircioğlu |
| collection | DOAJ |
| description | Abstract Background Many studies in radiomics are using feature selection methods to identify the most predictive features. At the same time, they employ cross-validation to estimate the performance of the developed models. However, if the feature selection is performed before the cross-validation, data leakage can occur, and the results can be biased. To measure the extent of this bias, we collected ten publicly available radiomics datasets and conducted two experiments. First, the models were developed by incorrectly applying the feature selection prior to cross-validation. Then, the same experiment was conducted by applying feature selection correctly within cross-validation to each fold. The resulting models were then evaluated against each other in terms of AUC-ROC, AUC-F1, and Accuracy. Results Applying the feature selection incorrectly prior to the cross-validation showed a bias of up to 0.15 in AUC-ROC, 0.29 in AUC-F1, and 0.17 in Accuracy. Conclusions Incorrect application of feature selection and cross-validation can lead to highly biased results for radiomic datasets. |
| first_indexed | 2024-12-13T16:24:15Z |
| format | Article |
| id | doaj.art-e05fc95b049c4fdc8f28b17d1566ac18 |
| institution | Directory Open Access Journal |
| issn | 1869-4101 |
| language | English |
| last_indexed | 2024-12-13T16:24:15Z |
| publishDate | 2021-11-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Insights into Imaging |
| spelling | doaj.art-e05fc95b049c4fdc8f28b17d1566ac182022-12-21T23:38:38ZengSpringerOpenInsights into Imaging1869-41012021-11-0112111010.1186/s13244-021-01115-1Measuring the bias of incorrect application of feature selection when using cross-validation in radiomicsAydin Demircioğlu0Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital EssenAbstract Background Many studies in radiomics are using feature selection methods to identify the most predictive features. At the same time, they employ cross-validation to estimate the performance of the developed models. However, if the feature selection is performed before the cross-validation, data leakage can occur, and the results can be biased. To measure the extent of this bias, we collected ten publicly available radiomics datasets and conducted two experiments. First, the models were developed by incorrectly applying the feature selection prior to cross-validation. Then, the same experiment was conducted by applying feature selection correctly within cross-validation to each fold. The resulting models were then evaluated against each other in terms of AUC-ROC, AUC-F1, and Accuracy. Results Applying the feature selection incorrectly prior to the cross-validation showed a bias of up to 0.15 in AUC-ROC, 0.29 in AUC-F1, and 0.17 in Accuracy. Conclusions Incorrect application of feature selection and cross-validation can lead to highly biased results for radiomic datasets.https://doi.org/10.1186/s13244-021-01115-1RadiomicsFeature selectionCross-validationBiasMachine learning |
| spellingShingle | Aydin Demircioğlu Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics Insights into Imaging Radiomics Feature selection Cross-validation Bias Machine learning |
| title | Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics |
| title_full | Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics |
| title_fullStr | Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics |
| title_full_unstemmed | Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics |
| title_short | Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics |
| title_sort | measuring the bias of incorrect application of feature selection when using cross validation in radiomics |
| topic | Radiomics Feature selection Cross-validation Bias Machine learning |
| url | https://doi.org/10.1186/s13244-021-01115-1 |
| work_keys_str_mv | AT aydindemircioglu measuringthebiasofincorrectapplicationoffeatureselectionwhenusingcrossvalidationinradiomics |