The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability

In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreem...

Full description

Bibliographic Details
Main Authors: Federico Cabitza, Andrea Campagner, Domenico Albano, Alberto Aliprandi, Alberto Bruno, Vito Chianca, Angelo Corazza, Francesco Di Pietto, Angelo Gambino, Salvatore Gitto, Carmelo Messina, Davide Orlandi, Luigi Pedone, Marcello Zappia, Luca Maria Sconfienza
Format: Article
Language:English
Published: MDPI AG 2020-06-01
Series:Applied Sciences
Subjects:
Online Access:https://www.mdpi.com/2076-3417/10/11/4014
_version_ 1827715249754603520
author Federico Cabitza
Andrea Campagner
Domenico Albano
Alberto Aliprandi
Alberto Bruno
Vito Chianca
Angelo Corazza
Francesco Di Pietto
Angelo Gambino
Salvatore Gitto
Carmelo Messina
Davide Orlandi
Luigi Pedone
Marcello Zappia
Luca Maria Sconfienza
author_facet Federico Cabitza
Andrea Campagner
Domenico Albano
Alberto Aliprandi
Alberto Bruno
Vito Chianca
Angelo Corazza
Francesco Di Pietto
Angelo Gambino
Salvatore Gitto
Carmelo Messina
Davide Orlandi
Luigi Pedone
Marcello Zappia
Luca Maria Sconfienza
author_sort Federico Cabitza
collection DOAJ
description In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters’ confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account.
first_indexed 2024-03-10T19:14:43Z
format Article
id doaj.art-215bf220cd494d83b64352bab2fc310e
institution Directory Open Access Journal
issn 2076-3417
language English
last_indexed 2024-03-10T19:14:43Z
publishDate 2020-06-01
publisher MDPI AG
record_format Article
series Applied Sciences
spelling doaj.art-215bf220cd494d83b64352bab2fc310e2023-11-20T03:27:14ZengMDPI AGApplied Sciences2076-34172020-06-011011401410.3390/app10114014The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification ReliabilityFederico Cabitza0Andrea Campagner1Domenico Albano2Alberto Aliprandi3Alberto Bruno4Vito Chianca5Angelo Corazza6Francesco Di Pietto7Angelo Gambino8Salvatore Gitto9Carmelo Messina10Davide Orlandi11Luigi Pedone12Marcello Zappia13Luca Maria Sconfienza14Department of Informatics, Sistemics and Communication (DISCo), University of Milano-Bicocca, 20126 Milano, ItalyDepartment of Informatics, Sistemics and Communication (DISCo), University of Milano-Bicocca, 20126 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyUnit of Radiology, Clinical Institutes Zucchi, 20900 Monza, ItalyDepartment of Biomedicine, Neuroscience and Advanced Diagnostics (BIND), University of Palermo, 90133 Palermo, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDiagnostic Imaging Department, Pineta Grande Hospital, 81030 Castel Volturno, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Biomedical Sciences for Health, Università degli Studi di Milano, 20122 Milano, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Radiology, Ospedale Evangelico Internazionale Genova, 16122 Genova, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyDepartment of Medicine and Health Sciences, University of Molise, 86100 Campobasso, ItalyIRCCS Istituto Ortopedico Galeazzi, 20161 Milano, ItalyIn this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters’ confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account.https://www.mdpi.com/2076-3417/10/11/4014inter-rater agreementreliabilityground truthmachine learningMRNetknee
spellingShingle Federico Cabitza
Andrea Campagner
Domenico Albano
Alberto Aliprandi
Alberto Bruno
Vito Chianca
Angelo Corazza
Francesco Di Pietto
Angelo Gambino
Salvatore Gitto
Carmelo Messina
Davide Orlandi
Luigi Pedone
Marcello Zappia
Luca Maria Sconfienza
The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
Applied Sciences
inter-rater agreement
reliability
ground truth
machine learning
MRNet
knee
title The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
title_full The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
title_fullStr The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
title_full_unstemmed The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
title_short The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability
title_sort elephant in the machine proposing a new metric of data reliability and its application to a medical case to assess classification reliability
topic inter-rater agreement
reliability
ground truth
machine learning
MRNet
knee
url https://www.mdpi.com/2076-3417/10/11/4014
work_keys_str_mv AT federicocabitza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT andreacampagner theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT domenicoalbano theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT albertoaliprandi theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT albertobruno theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT vitochianca theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT angelocorazza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT francescodipietto theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT angelogambino theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT salvatoregitto theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT carmelomessina theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT davideorlandi theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT luigipedone theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT marcellozappia theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT lucamariasconfienza theelephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT federicocabitza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT andreacampagner elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT domenicoalbano elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT albertoaliprandi elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT albertobruno elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT vitochianca elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT angelocorazza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT francescodipietto elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT angelogambino elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT salvatoregitto elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT carmelomessina elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT davideorlandi elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT luigipedone elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT marcellozappia elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability
AT lucamariasconfienza elephantinthemachineproposinganewmetricofdatareliabilityanditsapplicationtoamedicalcasetoassessclassificationreliability