Two contrasting data annotation paradigms for subjective NLP tasks
Labelled data is the foundation of most natural language processing tasks. However, labelling data is difficult and there often are diverse valid beliefs about what the correct data labels should be. So far, dataset creators have acknowledged annotator subjectivity, but rarely actively managed it in...
| Main Authors: | , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Association for Computational Linguistics
2022
|
| _version_ | 1826311916282183680 |
|---|---|
| author | Röttger, P Vidgen, B Hovy, D Pierrehumbert, JB |
| author_facet | Röttger, P Vidgen, B Hovy, D Pierrehumbert, JB |
| author_sort | Röttger, P |
| collection | OXFORD |
| description | Labelled data is the foundation of most natural language processing tasks. However, labelling data is difficult and there often are diverse valid beliefs about what the correct data labels should be. So far, dataset creators have acknowledged annotator subjectivity, but rarely actively managed it in the annotation process. This has led to partly-subjective datasets that fail to serve a clear downstream use. To address this issue, we propose two contrasting paradigms for data annotation. The descriptive paradigm encourages annotator subjectivity, whereas the prescriptive paradigm discourages it. Descriptive annotation allows for the surveying and modelling of different beliefs, whereas prescriptive annotation enables the training of models that consistently apply one belief. We discuss benefits and challenges in implementing both paradigms, and argue that dataset creators should explicitly aim for one or the other to facilitate the intended use of their dataset. Lastly, we conduct an annotation experiment using hate speech data that illustrates the contrast between the two paradigms. |
| first_indexed | 2024-03-07T08:18:20Z |
| format | Conference item |
| id | oxford-uuid:33ccc239-9bd7-402a-a2ab-4b8a850d705a |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T08:18:20Z |
| publishDate | 2022 |
| publisher | Association for Computational Linguistics |
| record_format | dspace |
| spelling | oxford-uuid:33ccc239-9bd7-402a-a2ab-4b8a850d705a2024-01-16T12:22:35ZTwo contrasting data annotation paradigms for subjective NLP tasksConference itemhttp://purl.org/coar/resource_type/c_5794uuid:33ccc239-9bd7-402a-a2ab-4b8a850d705aEnglishSymplectic ElementsAssociation for Computational Linguistics2022Röttger, PVidgen, BHovy, DPierrehumbert, JBLabelled data is the foundation of most natural language processing tasks. However, labelling data is difficult and there often are diverse valid beliefs about what the correct data labels should be. So far, dataset creators have acknowledged annotator subjectivity, but rarely actively managed it in the annotation process. This has led to partly-subjective datasets that fail to serve a clear downstream use. To address this issue, we propose two contrasting paradigms for data annotation. The descriptive paradigm encourages annotator subjectivity, whereas the prescriptive paradigm discourages it. Descriptive annotation allows for the surveying and modelling of different beliefs, whereas prescriptive annotation enables the training of models that consistently apply one belief. We discuss benefits and challenges in implementing both paradigms, and argue that dataset creators should explicitly aim for one or the other to facilitate the intended use of their dataset. Lastly, we conduct an annotation experiment using hate speech data that illustrates the contrast between the two paradigms. |
| spellingShingle | Röttger, P Vidgen, B Hovy, D Pierrehumbert, JB Two contrasting data annotation paradigms for subjective NLP tasks |
| title | Two contrasting data annotation paradigms for subjective NLP tasks |
| title_full | Two contrasting data annotation paradigms for subjective NLP tasks |
| title_fullStr | Two contrasting data annotation paradigms for subjective NLP tasks |
| title_full_unstemmed | Two contrasting data annotation paradigms for subjective NLP tasks |
| title_short | Two contrasting data annotation paradigms for subjective NLP tasks |
| title_sort | two contrasting data annotation paradigms for subjective nlp tasks |
| work_keys_str_mv | AT rottgerp twocontrastingdataannotationparadigmsforsubjectivenlptasks AT vidgenb twocontrastingdataannotationparadigmsforsubjectivenlptasks AT hovyd twocontrastingdataannotationparadigmsforsubjectivenlptasks AT pierrehumbertjb twocontrastingdataannotationparadigmsforsubjectivenlptasks |