The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreem...
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-06-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/10/11/4014 |
| _version_ | 1827715249754603520 |
|---|---|
| author | Federico Cabitza Andrea Campagner Domenico Albano Alberto Aliprandi Alberto Bruno Vito Chianca Angelo Corazza Francesco Di Pietto Angelo Gambino Salvatore Gitto Carmelo Messina Davide Orlandi Luigi Pedone Marcello Zappia Luca Maria Sconfienza |
| author_facet | Federico Cabitza Andrea Campagner Domenico Albano Alberto Aliprandi Alberto Bruno Vito Chianca Angelo Corazza Francesco Di Pietto Angelo Gambino Salvatore Gitto Carmelo Messina Davide Orlandi Luigi Pedone Marcello Zappia Luca Maria Sconfienza |
| author_sort | Federico Cabitza |
| collection | DOAJ |
| description | In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters’ confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account. |
| first_indexed | 2024-03-10T19:14:43Z |
| format | Article |
| id | doaj.art-215bf220cd494d83b64352bab2fc310e |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T19:14:43Z |
| publishDate | 2020-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-215bf220cd494d83b64352bab2fc310e2023-11-20T03:27:14ZengMDPI AGApplied Sciences2076-34172020-06-011011401410.3390/app10114014The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification ReliabilityFederico Cabitza0Andrea Campagner1Domenico Albano2Alberto Aliprandi3Alberto Bruno4Vito Chianca5Angelo Corazza6Francesco Di Pietto7Angelo Gambino8Salvatore Gitto9Carmelo Messina10Davide Orlandi11Luigi Pedone12Marcello Zappia13Luca Maria Sconfienza14Department of Informatics, Sistemics and Communication (DISCo), University of Milano-Bicocca, 20126 Milano, ItalyDepartment of Informatics, Sistemics and Communication (DISCo), University of Milano-Bicocca, 20126 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyUnit of Radiology, Clinical Institutes Zucchi, 20900 Monza, ItalyDepartment of Biomedicine, Neuroscience and Advanced Diagnostics (BIND), University of Palermo, 90133 Palermo, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDiagnostic Imaging Department, Pineta Grande Hospital, 81030 Castel Volturno, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Biomedical Sciences for Health, Università degli Studi di Milano, 20122 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Radiology, Ospedale Evangelico Internazionale Genova, 16122 Genova, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Medicine and Health Sciences, University of Molise, 86100 Campobasso, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyIn this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters’ confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account.https://www.mdpi.com/2076-3417/10/11/4014inter-rater agreementreliabilityground truthmachine learningMRNetknee |
| spellingShingle | Federico Cabitza Andrea Campagner Domenico Albano Alberto Aliprandi Alberto Bruno Vito Chianca Angelo Corazza Francesco Di Pietto Angelo Gambino Salvatore Gitto Carmelo Messina Davide Orlandi Luigi Pedone Marcello Zappia Luca Maria Sconfienza The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability Applied Sciences inter-rater agreement reliability ground truth machine learning MRNet knee |
| title | The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability |
| title_full | The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability |
| title_fullStr | The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability |
| title_full_unstemmed | The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability |
| title_short | The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability |
| title_sort | elephant in the machine proposing a new metric of data reliability and its application to a medical case to assess classification reliability |
| topic | inter-rater agreement reliability ground truth machine learning MRNet knee |
| url | https://www.mdpi.com/2076-3417/10/11/4014 |
| work_keys_str_mv | AT federicocabitza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT andreacampagner theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT domenicoalbano theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT albertoaliprandi theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT albertobruno theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT vitochianca theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT angelocorazza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT francescodipietto theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT angelogambino theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT salvatoregitto theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT carmelomessina theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT davideorlandi theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT luigipedone theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT marcellozappia theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT lucamariasconfienza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT federicocabitza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT andreacampagner elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT domenicoalbano elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT albertoaliprandi elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT albertobruno elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT vitochianca elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT angelocorazza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT francescodipietto elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT angelogambino elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT salvatoregitto elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT carmelomessina elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT davideorlandi elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT luigipedone elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT marcellozappia elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability AT lucamariasconfienza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability |