Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization
Facial expression recognition (FER) has been extensively studied in various applications over the past few years. However, in real facial expression datasets, labels can become noisy due to the ambiguity of expressions, the similarity between classes, and the subjectivity of annotators. These noisy...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10345539/ |
| _version_ | 1797376280503517184 |
|---|---|
| author | Jihyun Kim Junehyoung Kwon Mihyeon Kim Eunju Lee Youngbin Kim |
| author_facet | Jihyun Kim Junehyoung Kwon Mihyeon Kim Eunju Lee Youngbin Kim |
| author_sort | Jihyun Kim |
| collection | DOAJ |
| description | Facial expression recognition (FER) has been extensively studied in various applications over the past few years. However, in real facial expression datasets, labels can become noisy due to the ambiguity of expressions, the similarity between classes, and the subjectivity of annotators. These noisy labels negatively affect FER and significantly reduce classification performance. In previous methods, overfitting can occur as the noise ratio increases. To solve this problem, we propose the split and merge consistency regularization (SMEC) method that is robust to noisy labels by examining various image regions rather than just one part of facial expression images without negatively affecting the meaning. We split facial expression images into two images and input them into the backbone network to extract class activation maps (CAMs). This approach merges two CAMs and improves robustness to noisy labels by normalizing the consistency between the CAM of the original image and the merged CAM. The proposed SMEC method aims to improve FER performance and robustness against highly noisy labels by preventing the model from focusing on only a single part without losing the semantics of the facial expression images. The SMEC method demonstrates robust performance over state-of-the-art noisy label FER models on an unbalanced facial expression dataset called the real-world affective faces database (RAF-DB) regarding class-wise accuracy for clean and noisy labels, even at severe noise rates of 40% to 60%. |
| first_indexed | 2024-03-08T19:36:15Z |
| format | Article |
| id | doaj.art-481e228e61434c6789aed54ca23a18a6 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-08T19:36:15Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-481e228e61434c6789aed54ca23a18a62023-12-26T00:10:17ZengIEEEIEEE Access2169-35362023-01-011114049614050510.1109/ACCESS.2023.333976310345539Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency RegularizationJihyun Kim0Junehyoung Kwon1Mihyeon Kim2Eunju Lee3https://orcid.org/0000-0002-6571-0156Youngbin Kim4https://orcid.org/0000-0002-2114-0120Department of Artificial Intelligence, Chung-Ang University, Dongjak, South KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak, South KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak, South KoreaDepartment of Imaging Science, Multimedia and Film, Chung-Ang University, Dongjak, South KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak, South KoreaFacial expression recognition (FER) has been extensively studied in various applications over the past few years. However, in real facial expression datasets, labels can become noisy due to the ambiguity of expressions, the similarity between classes, and the subjectivity of annotators. These noisy labels negatively affect FER and significantly reduce classification performance. In previous methods, overfitting can occur as the noise ratio increases. To solve this problem, we propose the split and merge consistency regularization (SMEC) method that is robust to noisy labels by examining various image regions rather than just one part of facial expression images without negatively affecting the meaning. We split facial expression images into two images and input them into the backbone network to extract class activation maps (CAMs). This approach merges two CAMs and improves robustness to noisy labels by normalizing the consistency between the CAM of the original image and the merged CAM. The proposed SMEC method aims to improve FER performance and robustness against highly noisy labels by preventing the model from focusing on only a single part without losing the semantics of the facial expression images. The SMEC method demonstrates robust performance over state-of-the-art noisy label FER models on an unbalanced facial expression dataset called the real-world affective faces database (RAF-DB) regarding class-wise accuracy for clean and noisy labels, even at severe noise rates of 40% to 60%.https://ieeexplore.ieee.org/document/10345539/Consistency regularizationdeep learningfacial expression recognitionimage classificationnoisy label learning |
| spellingShingle | Jihyun Kim Junehyoung Kwon Mihyeon Kim Eunju Lee Youngbin Kim Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization IEEE Access Consistency regularization deep learning facial expression recognition image classification noisy label learning |
| title | Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization |
| title_full | Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization |
| title_fullStr | Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization |
| title_full_unstemmed | Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization |
| title_short | Enhancing Noisy Label Facial Expression Recognition With Split and Merge Consistency Regularization |
| title_sort | enhancing noisy label facial expression recognition with split and merge consistency regularization |
| topic | Consistency regularization deep learning facial expression recognition image classification noisy label learning |
| url | https://ieeexplore.ieee.org/document/10345539/ |
| work_keys_str_mv | AT jihyunkim enhancingnoisylabelfacialexpressionrecognitionwithsplitandmergeconsistencyregularization AT junehyoungkwon enhancingnoisylabelfacialexpressionrecognitionwithsplitandmergeconsistencyregularization AT mihyeonkim enhancingnoisylabelfacialexpressionrecognitionwithsplitandmergeconsistencyregularization AT eunjulee enhancingnoisylabelfacialexpressionrecognitionwithsplitandmergeconsistencyregularization AT youngbinkim enhancingnoisylabelfacialexpressionrecognitionwithsplitandmergeconsistencyregularization |