Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions

Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the infl...

Full description

Bibliographic Details
Main Authors: Quentin Noirhomme, Damien Lesenfants, Francisco Gomez, Andrea Soddu, Jessica Schrouff, Gaëtan Garraux, André Luxen, Christophe Phillips, Steven Laureys
Format: Article
Language:English
Published: Elsevier 2014-01-01
Series:NeuroImage: Clinical
Subjects:
Online Access:http://www.sciencedirect.com/science/article/pii/S2213158214000485
_version_ 1818518803069272064
author Quentin Noirhomme
Damien Lesenfants
Francisco Gomez
Andrea Soddu
Jessica Schrouff
Gaëtan Garraux
André Luxen
Christophe Phillips
Steven Laureys
author_facet Quentin Noirhomme
Damien Lesenfants
Francisco Gomez
Andrea Soddu
Jessica Schrouff
Gaëtan Garraux
André Luxen
Christophe Phillips
Steven Laureys
author_sort Quentin Noirhomme
collection DOAJ
description Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain–computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.
first_indexed 2024-12-11T01:15:15Z
format Article
id doaj.art-01e67d73dcce40cd8a76e9664d0f8416
institution Directory Open Access Journal
issn 2213-1582
language English
last_indexed 2024-12-11T01:15:15Z
publishDate 2014-01-01
publisher Elsevier
record_format Article
series NeuroImage: Clinical
spelling doaj.art-01e67d73dcce40cd8a76e9664d0f84162022-12-22T01:25:54ZengElsevierNeuroImage: Clinical2213-15822014-01-014C68769410.1016/j.nicl.2014.04.004Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictionsQuentin Noirhomme0Damien Lesenfants1Francisco Gomez2Andrea Soddu3Jessica Schrouff4Gaëtan Garraux5André Luxen6Christophe Phillips7Steven Laureys8Cyclotron Research Centre, University of Liège, Liège, BelgiumCyclotron Research Centre, University of Liège, Liège, BelgiumComplexus Group, Computer Science Department, Universidad Central de Colombia, Bogotá, ColombiaDepartment of Physics & Astronomy, Brain and Mind Institute, University of Western Ontario, London, ON, CanadaCyclotron Research Centre, University of Liège, Liège, BelgiumCyclotron Research Centre, University of Liège, Liège, BelgiumCyclotron Research Centre, University of Liège, Liège, BelgiumCyclotron Research Centre, University of Liège, Liège, BelgiumCyclotron Research Centre, University of Liège, Liège, BelgiumMultivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain–computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.http://www.sciencedirect.com/science/article/pii/S2213158214000485classificationcross-validationbinomialpermutation test
spellingShingle Quentin Noirhomme
Damien Lesenfants
Francisco Gomez
Andrea Soddu
Jessica Schrouff
Gaëtan Garraux
André Luxen
Christophe Phillips
Steven Laureys
Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
NeuroImage: Clinical
classification
cross-validation
binomial
permutation test
title Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
title_full Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
title_fullStr Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
title_full_unstemmed Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
title_short Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions
title_sort biased binomial assessment of cross validated estimation of classification accuracies illustrated in diagnosis predictions
topic classification
cross-validation
binomial
permutation test
url http://www.sciencedirect.com/science/article/pii/S2213158214000485
work_keys_str_mv AT quentinnoirhomme biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT damienlesenfants biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT franciscogomez biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT andreasoddu biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT jessicaschrouff biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT gaetangarraux biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT andreluxen biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT christophephillips biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions
AT stevenlaureys biasedbinomialassessmentofcrossvalidatedestimationofclassificationaccuraciesillustratedindiagnosispredictions