Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data
As governments and industry turn to increased use of automated decision systems, it becomes essential to consider how closely such systems can reproduce human judgment. We identify a core potential failure, finding that annotators label objects differently depending on whether they are being asked a...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | en_US |
| Published: |
American Association for the Advancement of Science (AAAS)
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/1721.1/153492 |
| _version_ | 1811073065818259456 |
|---|---|
| author | Balagopalan, Aparna Madras, David Yang, David H. Hadfield-Menell, Dylan Hadfield, Gillian K. Ghassemi, Marzyeh |
| author_facet | Balagopalan, Aparna Madras, David Yang, David H. Hadfield-Menell, Dylan Hadfield, Gillian K. Ghassemi, Marzyeh |
| author_sort | Balagopalan, Aparna |
| collection | MIT |
| description | As governments and industry turn to increased use of automated decision systems, it becomes essential to consider how closely such systems can reproduce human judgment. We identify a core potential failure, finding that annotators label objects differently depending on whether they are being asked a factual question or a normative question. This challenges a natural assumption maintained in many standard machine-learning (ML) data acquisition procedures: that there is no difference between predicting the factual classification of an object and an exercise of judgment about whether an object violates a rule premised on those facts. We find that using factual labels to train models intended for normative judgments introduces a notable measurement error. We show that models trained using factual labels yield significantly different judgments than those trained using normative labels and that the impact of this effect on model performance can exceed that of other factors (e.g., dataset size) that routinely attract attention from ML researchers and practitioners. |
| first_indexed | 2024-09-23T09:28:04Z |
| format | Article |
| id | mit-1721.1/153492 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T09:28:04Z |
| publishDate | 2024 |
| publisher | American Association for the Advancement of Science (AAAS) |
| record_format | dspace |
| spelling | mit-1721.1/1534922024-02-10T04:02:58Z Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data Balagopalan, Aparna Madras, David Yang, David H. Hadfield-Menell, Dylan Hadfield, Gillian K. Ghassemi, Marzyeh Multidisciplinary As governments and industry turn to increased use of automated decision systems, it becomes essential to consider how closely such systems can reproduce human judgment. We identify a core potential failure, finding that annotators label objects differently depending on whether they are being asked a factual question or a normative question. This challenges a natural assumption maintained in many standard machine-learning (ML) data acquisition procedures: that there is no difference between predicting the factual classification of an object and an exercise of judgment about whether an object violates a rule premised on those facts. We find that using factual labels to train models intended for normative judgments introduces a notable measurement error. We show that models trained using factual labels yield significantly different judgments than those trained using normative labels and that the impact of this effect on model performance can exceed that of other factors (e.g., dataset size) that routinely attract attention from ML researchers and practitioners. 2024-02-09T20:48:29Z 2024-02-09T20:48:29Z 2023-05-12 Article http://purl.org/eprint/type/JournalArticle 2375-2548 https://hdl.handle.net/1721.1/153492 Aparna Balagopalan et al. ,Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data.Sci. Adv.9,eabq0701(2023). en_US 10.1126/sciadv.abq0701 Creative Commons Attribution An error occurred on the license name. https://creativecommons.org/licenses/by/4.0/ application/pdf American Association for the Advancement of Science (AAAS) American Association for the Advancement of Science |
| spellingShingle | Multidisciplinary Balagopalan, Aparna Madras, David Yang, David H. Hadfield-Menell, Dylan Hadfield, Gillian K. Ghassemi, Marzyeh Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title | Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title_full | Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title_fullStr | Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title_full_unstemmed | Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title_short | Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data |
| title_sort | judging facts judging norms training machine learning models to judge humans requires a modified approach to labeling data |
| topic | Multidisciplinary |
| url | https://hdl.handle.net/1721.1/153492 |
| work_keys_str_mv | AT balagopalanaparna judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata AT madrasdavid judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata AT yangdavidh judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata AT hadfieldmenelldylan judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata AT hadfieldgilliank judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata AT ghassemimarzyeh judgingfactsjudgingnormstrainingmachinelearningmodelstojudgehumansrequiresamodifiedapproachtolabelingdata |