The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment
Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summ...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9440903/ |
| _version_ | 1819092750543355904 |
|---|---|
| author | Davide Chicco Matthijs J. Warrens Giuseppe Jurman |
| author_facet | Davide Chicco Matthijs J. Warrens Giuseppe Jurman |
| author_sort | Davide Chicco |
| collection | DOAJ |
| description | Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F<sub>1</sub> score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications. |
| first_indexed | 2024-12-21T23:00:35Z |
| format | Article |
| id | doaj.art-91ef0226db1248cf90447c75247d23bc |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-21T23:00:35Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-91ef0226db1248cf90447c75247d23bc2022-12-21T18:47:18ZengIEEEIEEE Access2169-35362021-01-019783687838110.1109/ACCESS.2021.30840509440903The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification AssessmentDavide Chicco0https://orcid.org/0000-0001-9655-7142Matthijs J. Warrens1https://orcid.org/0000-0002-7302-640XGiuseppe Jurman2https://orcid.org/0000-0002-2705-5728Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, CanadaGroningen Institute for Educational Research, University of Groningen, Groningen, The NetherlandsData Science for Health Unit, Fondazione Bruno Kessler, Trento, ItalyEven if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F<sub>1</sub> score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications.https://ieeexplore.ieee.org/document/9440903/Matthews correlation coefficientCohen’s Kappabinary classificationconfusion matrixsupervised machine learningBrier score |
| spellingShingle | Davide Chicco Matthijs J. Warrens Giuseppe Jurman The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment IEEE Access Matthews correlation coefficient Cohen’s Kappa binary classification confusion matrix supervised machine learning Brier score |
| title | The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment |
| title_full | The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment |
| title_fullStr | The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment |
| title_full_unstemmed | The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment |
| title_short | The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment |
| title_sort | matthews correlation coefficient mcc is more informative than cohen x2019 s kappa and brier score in binary classification assessment |
| topic | Matthews correlation coefficient Cohen’s Kappa binary classification confusion matrix supervised machine learning Brier score |
| url | https://ieeexplore.ieee.org/document/9440903/ |
| work_keys_str_mv | AT davidechicco thematthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment AT matthijsjwarrens thematthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment AT giuseppejurman thematthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment AT davidechicco matthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment AT matthijsjwarrens matthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment AT giuseppejurman matthewscorrelationcoefficientmccismoreinformativethancohenx2019skappaandbrierscoreinbinaryclassificationassessment |