Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement
Learning with noisy labels is one of the most practical but challenging tasks in deep learning. One promising way to treat noisy labels is to use the small-loss trick based on the memorization effect, that is, clean and noisy samples are identified by observing the network’s loss during t...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9568980/ |
| _version_ | 1818587233273249792 |
|---|---|
| author | Hiroshi Takeda Soh Yoshida Mitsuji Muneyasu |
| author_facet | Hiroshi Takeda Soh Yoshida Mitsuji Muneyasu |
| author_sort | Hiroshi Takeda |
| collection | DOAJ |
| description | Learning with noisy labels is one of the most practical but challenging tasks in deep learning. One promising way to treat noisy labels is to use the small-loss trick based on the memorization effect, that is, clean and noisy samples are identified by observing the network’s loss during training. Co-teaching+ is a state-of-the-art method that simultaneously trains two networks with small-loss selection using the “update by disagreement” strategy; however, it suffers from the problem that the selected samples tend to become noisy as the number of iterations increases. This phenomenon means that clean small-loss samples will be biased toward agreement data, which is the set of samples for which the two networks have the same prediction. This paper proposes an adaptive sample selection method to train deep neural networks robustly and prevent noise contamination in the disagreement strategy. Specifically, the proposed method calculates the threshold of the small-loss criterion by considering the loss distribution of the whole batch at each iteration. Then, the network is backpropagated by extracting samples below this threshold from the disagreement data. Combining the disagreement and agreement data of the two networks can suppress the degradation of the true-label rate of training data in a mini batch. Experiments were conducted using five commonly used benchmarks, MNIST, CIFAR-10, CIFAR-100, NEWS, and T-ImageNet to verify the robustness of the proposed method to noisy labels. The results show the proposed method improves generalization performance in an image classification task with simulated noise rates of up to 50%. |
| first_indexed | 2024-12-16T09:05:36Z |
| format | Article |
| id | doaj.art-1576c61166b146aa925b8065e95c97bf |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-16T09:05:36Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-1576c61166b146aa925b8065e95c97bf2022-12-21T22:37:04ZengIEEEIEEE Access2169-35362021-01-01914113114114310.1109/ACCESS.2021.31195829568980Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With DisagreementHiroshi Takeda0https://orcid.org/0000-0003-0713-6878Soh Yoshida1https://orcid.org/0000-0003-0237-7461Mitsuji Muneyasu2https://orcid.org/0000-0002-4492-5991Graduate School of Science and Engineering, Kansai University, Suita-shi, Osaka, JapanFaculty of Engineering Science, Kansai University, Suita-shi, Osaka, JapanFaculty of Engineering Science, Kansai University, Suita-shi, Osaka, JapanLearning with noisy labels is one of the most practical but challenging tasks in deep learning. One promising way to treat noisy labels is to use the small-loss trick based on the memorization effect, that is, clean and noisy samples are identified by observing the network’s loss during training. Co-teaching+ is a state-of-the-art method that simultaneously trains two networks with small-loss selection using the “update by disagreement” strategy; however, it suffers from the problem that the selected samples tend to become noisy as the number of iterations increases. This phenomenon means that clean small-loss samples will be biased toward agreement data, which is the set of samples for which the two networks have the same prediction. This paper proposes an adaptive sample selection method to train deep neural networks robustly and prevent noise contamination in the disagreement strategy. Specifically, the proposed method calculates the threshold of the small-loss criterion by considering the loss distribution of the whole batch at each iteration. Then, the network is backpropagated by extracting samples below this threshold from the disagreement data. Combining the disagreement and agreement data of the two networks can suppress the degradation of the true-label rate of training data in a mini batch. Experiments were conducted using five commonly used benchmarks, MNIST, CIFAR-10, CIFAR-100, NEWS, and T-ImageNet to verify the robustness of the proposed method to noisy labels. The results show the proposed method improves generalization performance in an image classification task with simulated noise rates of up to 50%.https://ieeexplore.ieee.org/document/9568980/Deep neural networklearning with noisy labelsimage classificationco-teaching |
| spellingShingle | Hiroshi Takeda Soh Yoshida Mitsuji Muneyasu Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement IEEE Access Deep neural network learning with noisy labels image classification co-teaching |
| title | Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement |
| title_full | Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement |
| title_fullStr | Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement |
| title_full_unstemmed | Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement |
| title_short | Training Robust Deep Neural Networks on Noisy Labels Using Adaptive Sample Selection With Disagreement |
| title_sort | training robust deep neural networks on noisy labels using adaptive sample selection with disagreement |
| topic | Deep neural network learning with noisy labels image classification co-teaching |
| url | https://ieeexplore.ieee.org/document/9568980/ |
| work_keys_str_mv | AT hiroshitakeda trainingrobustdeepneuralnetworksonnoisylabelsusingadaptivesampleselectionwithdisagreement AT sohyoshida trainingrobustdeepneuralnetworksonnoisylabelsusingadaptivesampleselectionwithdisagreement AT mitsujimuneyasu trainingrobustdeepneuralnetworksonnoisylabelsusingadaptivesampleselectionwithdisagreement |