On reliability of annotations in contextual emotion imagery
Abstract We documented the relabeling process for a subset of a renowned database for emotion-in-context recognition, with the aim of promoting reliability in final labels. To this end, emotion categories were organized into eight groups, while a large number of participants was requested for taggin...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-08-01
|
| Series: | Scientific Data |
| Online Access: | https://doi.org/10.1038/s41597-023-02435-1 |
| _version_ | 1797453929158541312 |
|---|---|
| author | Carlos A. Martínez-Miwa Mario Castelán |
| author_facet | Carlos A. Martínez-Miwa Mario Castelán |
| author_sort | Carlos A. Martínez-Miwa |
| collection | DOAJ |
| description | Abstract We documented the relabeling process for a subset of a renowned database for emotion-in-context recognition, with the aim of promoting reliability in final labels. To this end, emotion categories were organized into eight groups, while a large number of participants was requested for tagging. A strict control strategy was performed along the experiments, whose duration was 13.45 minutes average per day. Annotators were free to participate in any of the daily experiments (the average number of participants was 28), and a Z-Score filtering technique was implemented to keep trustworthiness of annotations. As a result, the value of the agreement parameter Fleiss’ Kapa increasingly varied from slight to almost perfect, revealing a coherent diversity of the experiments. Our results support the hypothesis that a small number of categories and a large number of voters benefit reliability of annotations in contextual emotion imagery. |
| first_indexed | 2024-03-09T15:29:57Z |
| format | Article |
| id | doaj.art-27b840f2653748afa0c081a22e1edca9 |
| institution | Directory Open Access Journal |
| issn | 2052-4463 |
| language | English |
| last_indexed | 2024-03-09T15:29:57Z |
| publishDate | 2023-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Data |
| spelling | doaj.art-27b840f2653748afa0c081a22e1edca92023-11-26T12:18:58ZengNature PortfolioScientific Data2052-44632023-08-0110111210.1038/s41597-023-02435-1On reliability of annotations in contextual emotion imageryCarlos A. Martínez-Miwa0Mario Castelán1Robótica y Manufactura Avanzada, Centro de Investigación y de Estudios Avanzados del Instituto Politécnico NacionalRobótica y Manufactura Avanzada, Centro de Investigación y de Estudios Avanzados del Instituto Politécnico NacionalAbstract We documented the relabeling process for a subset of a renowned database for emotion-in-context recognition, with the aim of promoting reliability in final labels. To this end, emotion categories were organized into eight groups, while a large number of participants was requested for tagging. A strict control strategy was performed along the experiments, whose duration was 13.45 minutes average per day. Annotators were free to participate in any of the daily experiments (the average number of participants was 28), and a Z-Score filtering technique was implemented to keep trustworthiness of annotations. As a result, the value of the agreement parameter Fleiss’ Kapa increasingly varied from slight to almost perfect, revealing a coherent diversity of the experiments. Our results support the hypothesis that a small number of categories and a large number of voters benefit reliability of annotations in contextual emotion imagery.https://doi.org/10.1038/s41597-023-02435-1 |
| spellingShingle | Carlos A. Martínez-Miwa Mario Castelán On reliability of annotations in contextual emotion imagery Scientific Data |
| title | On reliability of annotations in contextual emotion imagery |
| title_full | On reliability of annotations in contextual emotion imagery |
| title_fullStr | On reliability of annotations in contextual emotion imagery |
| title_full_unstemmed | On reliability of annotations in contextual emotion imagery |
| title_short | On reliability of annotations in contextual emotion imagery |
| title_sort | on reliability of annotations in contextual emotion imagery |
| url | https://doi.org/10.1038/s41597-023-02435-1 |
| work_keys_str_mv | AT carlosamartinezmiwa onreliabilityofannotationsincontextualemotionimagery AT mariocastelan onreliabilityofannotationsincontextualemotionimagery |