Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. St...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/7/3518 |
| _version_ | 1797607087024373760 |
|---|---|
| author | Jenniffer Carolina Triana-Martinez Julian Gil-González Jose A. Fernandez-Gallego Andrés Marino Álvarez-Meza Cesar German Castellanos-Dominguez |
| author_facet | Jenniffer Carolina Triana-Martinez Julian Gil-González Jose A. Fernandez-Gallego Andrés Marino Álvarez-Meza Cesar German Castellanos-Dominguez |
| author_sort | Jenniffer Carolina Triana-Martinez |
| collection | DOAJ |
| description | Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation. |
| first_indexed | 2024-03-11T05:25:21Z |
| format | Article |
| id | doaj.art-f2af93c79b4948c9ba5a4126634a127a |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T05:25:21Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-f2af93c79b4948c9ba5a4126634a127a2023-11-17T17:33:44ZengMDPI AGSensors1424-82202023-03-01237351810.3390/s23073518Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators ClassificationJenniffer Carolina Triana-Martinez0Julian Gil-González1Jose A. Fernandez-Gallego2Andrés Marino Álvarez-Meza3Cesar German Castellanos-Dominguez4Signal Processing and Recognition Group, Universidad Nacional de Colombia, Manizales 170003, ColombiaDepartment of Electronics and Computer Science, Pontificia Universidad Javeriana Cali, Cali 760031, ColombiaPrograma de Ingeniería Electrónica, Facultad de Ingeniería, Universidad de Ibagué, Ibagué 730001, ColombiaSignal Processing and Recognition Group, Universidad Nacional de Colombia, Manizales 170003, ColombiaSignal Processing and Recognition Group, Universidad Nacional de Colombia, Manizales 170003, ColombiaSupervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation.https://www.mdpi.com/1424-8220/23/7/3518deep learningmultiple annotatorschained approachgeneralized cross-entropyclassification |
| spellingShingle | Jenniffer Carolina Triana-Martinez Julian Gil-González Jose A. Fernandez-Gallego Andrés Marino Álvarez-Meza Cesar German Castellanos-Dominguez Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification Sensors deep learning multiple annotators chained approach generalized cross-entropy classification |
| title | Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification |
| title_full | Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification |
| title_fullStr | Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification |
| title_full_unstemmed | Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification |
| title_short | Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification |
| title_sort | chained deep learning using generalized cross entropy for multiple annotators classification |
| topic | deep learning multiple annotators chained approach generalized cross-entropy classification |
| url | https://www.mdpi.com/1424-8220/23/7/3518 |
| work_keys_str_mv | AT jenniffercarolinatrianamartinez chaineddeeplearningusinggeneralizedcrossentropyformultipleannotatorsclassification AT juliangilgonzalez chaineddeeplearningusinggeneralizedcrossentropyformultipleannotatorsclassification AT joseafernandezgallego chaineddeeplearningusinggeneralizedcrossentropyformultipleannotatorsclassification AT andresmarinoalvarezmeza chaineddeeplearningusinggeneralizedcrossentropyformultipleannotatorsclassification AT cesargermancastellanosdominguez chaineddeeplearningusinggeneralizedcrossentropyformultipleannotatorsclassification |