Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example
The stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluat...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-10-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/8/10/1741 |
| _version_ | 1797551341220921344 |
|---|---|
| author | Andra Ciocan Nadim Al Hajjar Florin Graur Valentin C. Oprea Răzvan A. Ciocan Sorana D. Bolboacă |
| author_facet | Andra Ciocan Nadim Al Hajjar Florin Graur Valentin C. Oprea Răzvan A. Ciocan Sorana D. Bolboacă |
| author_sort | Andra Ciocan |
| collection | DOAJ |
| description | The stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluated as predictors for metastasis in patients with colorectal cancer, was investigated. Data belonging to patients admitted with the diagnosis of colorectal cancer from January 2014 until September 2019 in a single hospital were used. There were 1688 patients eligible for the study, 418 in the metastatic stage. All investigated inflammatory ratios proved to be significant classification models on both the full models and on cross-validations (AUCs > 0.05). High variability of the cut-off values was observed in the unrestricted and restricted split (full models: 4.255 for NLR, 2.745 for dNLR and 255.56 for PLR; random splits: cut-off from 3.215 to 5.905 for NLR, from 2.625 to 3.575 for dNLR and from 134.67 to 335.9 for PLR), but with no effect on the models characteristics or performances. The investigated biomarkes proved limited value as predictors for metastasis (AUCs < 0.8), with largely sensitivity and specificity (from 33.3% to 79.2% for the full model and 29.1% to 82.7% in the restricted splits). Our results showed that a simple random split of observations, weighting or not the patients with and whithout metastasis, in a ROC analysis assures the performances similar to the full model, if at least 70% of the available population is included in the study. |
| first_indexed | 2024-03-10T15:44:15Z |
| format | Article |
| id | doaj.art-49495e6828b04da781fa61d0705e2c2e |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T15:44:15Z |
| publishDate | 2020-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-49495e6828b04da781fa61d0705e2c2e2023-11-20T16:35:54ZengMDPI AGMathematics2227-73902020-10-01810174110.3390/math8101741Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by ExampleAndra Ciocan0Nadim Al Hajjar1Florin Graur2Valentin C. Oprea3Răzvan A. Ciocan4Sorana D. Bolboacă5Department of Medical Informatics and Biostatistics, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Louis Pasteur Street, No. 6, 400349 Cluj-Napoca, Romania“Prof. Dr. Octavian Fodor” Regional Institute of Gastroenterology and Hepatology Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania“Prof. Dr. Octavian Fodor” Regional Institute of Gastroenterology and Hepatology Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania“Dr. Constantin Papilian” Military Emergency Hospital Cluj-Napoca, General Traian Moșoiu Street, No. 22, 400132 Cluj-Napoca, RomaniaDepartment of Medical Skills—Human Sciences, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Marinescu Street, No. 23, 400337 Cluj-Napoca, RomaniaDepartment of Medical Informatics and Biostatistics, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Louis Pasteur Street, No. 6, 400349 Cluj-Napoca, RomaniaThe stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluated as predictors for metastasis in patients with colorectal cancer, was investigated. Data belonging to patients admitted with the diagnosis of colorectal cancer from January 2014 until September 2019 in a single hospital were used. There were 1688 patients eligible for the study, 418 in the metastatic stage. All investigated inflammatory ratios proved to be significant classification models on both the full models and on cross-validations (AUCs > 0.05). High variability of the cut-off values was observed in the unrestricted and restricted split (full models: 4.255 for NLR, 2.745 for dNLR and 255.56 for PLR; random splits: cut-off from 3.215 to 5.905 for NLR, from 2.625 to 3.575 for dNLR and from 134.67 to 335.9 for PLR), but with no effect on the models characteristics or performances. The investigated biomarkes proved limited value as predictors for metastasis (AUCs < 0.8), with largely sensitivity and specificity (from 33.3% to 79.2% for the full model and 29.1% to 82.7% in the restricted splits). Our results showed that a simple random split of observations, weighting or not the patients with and whithout metastasis, in a ROC analysis assures the performances similar to the full model, if at least 70% of the available population is included in the study.https://www.mdpi.com/2227-7390/8/10/1741receiver operating characteristicarea under the curvemodels performancesbiomarkersmetastasis |
| spellingShingle | Andra Ciocan Nadim Al Hajjar Florin Graur Valentin C. Oprea Răzvan A. Ciocan Sorana D. Bolboacă Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example Mathematics receiver operating characteristic area under the curve models performances biomarkers metastasis |
| title | Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example |
| title_full | Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example |
| title_fullStr | Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example |
| title_full_unstemmed | Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example |
| title_short | Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example |
| title_sort | receiver operating characteristic prediction for classification performances in cross validation by example |
| topic | receiver operating characteristic area under the curve models performances biomarkers metastasis |
| url | https://www.mdpi.com/2227-7390/8/10/1741 |
| work_keys_str_mv | AT andraciocan receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample AT nadimalhajjar receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample AT floringraur receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample AT valentincoprea receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample AT razvanaciocan receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample AT soranadbolboaca receiveroperatingcharacteristicpredictionforclassificationperformancesincrossvalidationbyexample |