Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image
Adversarial examples easily mislead vision systems based on deep neural networks (DNNs) trained with softmax cross entropy (SCE) loss. The vulnerability of DNN comes from the fact that SCE drives DNNs to fit on the training examples, whereas the resultant feature distributions between the training a...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/10/19/3716 |
| _version_ | 1797478051364208640 |
|---|---|
| author | Bingbing Song Ruxin Wang Wei He Wei Zhou |
| author_facet | Bingbing Song Ruxin Wang Wei He Wei Zhou |
| author_sort | Bingbing Song |
| collection | DOAJ |
| description | Adversarial examples easily mislead vision systems based on deep neural networks (DNNs) trained with softmax cross entropy (SCE) loss. The vulnerability of DNN comes from the fact that SCE drives DNNs to fit on the training examples, whereas the resultant feature distributions between the training and adversarial examples are unfortunately misaligned. Several state-of-the-art methods start from improving the inter-class separability of training examples by modifying loss functions, where we argue that the adversarial examples are ignored, thus resulting in a limited robustness to adversarial attacks. In this paper, we exploited the inference region, which inspired us to apply margin-like inference information to SCE, resulting in a novel inference-softmax cross entropy (I-SCE) loss, which is intuitively appealing and interpretable. The inference information guarantees that it is difficult for neural networks to cross the decision boundary under an adversarial attack, and guarantees both the inter-class separability and the improved generalization to adversarial examples, which was further demonstrated and proved under the min-max framework. Extensive experiments show that the DNN models trained with the proposed I-SCE loss achieve a superior performance and robustness over the state-of-the-arts under different prevalent adversarial attacks; for example, the accuracy of I-SCE is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>63%</mn></mrow></semantics></math></inline-formula> higher than SCE under the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi><msubsup><mi>D</mi><mrow><mn>50</mn></mrow><mrow><mi>u</mi><mi>n</mi></mrow></msubsup></mrow></semantics></math></inline-formula> attack on the MNIST dataset. These experiments also show that the inference region can effectively solve the misaligned distribution. |
| first_indexed | 2024-03-09T21:26:33Z |
| format | Article |
| id | doaj.art-64be4712ee024a729356982251db4a7a |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-09T21:26:33Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-64be4712ee024a729356982251db4a7a2023-11-23T21:06:18ZengMDPI AGMathematics2227-73902022-10-011019371610.3390/math10193716Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of ImageBingbing Song0Ruxin Wang1Wei He2Wei Zhou3School of Information Science and Engineering, Yunnan University, Kunming 650091, ChinaPiolet School of Software, Yunnan University, Kunming 650091, ChinaPiolet School of Software, Yunnan University, Kunming 650091, ChinaPiolet School of Software, Yunnan University, Kunming 650091, ChinaAdversarial examples easily mislead vision systems based on deep neural networks (DNNs) trained with softmax cross entropy (SCE) loss. The vulnerability of DNN comes from the fact that SCE drives DNNs to fit on the training examples, whereas the resultant feature distributions between the training and adversarial examples are unfortunately misaligned. Several state-of-the-art methods start from improving the inter-class separability of training examples by modifying loss functions, where we argue that the adversarial examples are ignored, thus resulting in a limited robustness to adversarial attacks. In this paper, we exploited the inference region, which inspired us to apply margin-like inference information to SCE, resulting in a novel inference-softmax cross entropy (I-SCE) loss, which is intuitively appealing and interpretable. The inference information guarantees that it is difficult for neural networks to cross the decision boundary under an adversarial attack, and guarantees both the inter-class separability and the improved generalization to adversarial examples, which was further demonstrated and proved under the min-max framework. Extensive experiments show that the DNN models trained with the proposed I-SCE loss achieve a superior performance and robustness over the state-of-the-arts under different prevalent adversarial attacks; for example, the accuracy of I-SCE is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>63%</mn></mrow></semantics></math></inline-formula> higher than SCE under the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>P</mi><mi>G</mi><msubsup><mi>D</mi><mrow><mn>50</mn></mrow><mrow><mi>u</mi><mi>n</mi></mrow></msubsup></mrow></semantics></math></inline-formula> attack on the MNIST dataset. These experiments also show that the inference region can effectively solve the misaligned distribution.https://www.mdpi.com/2227-7390/10/19/3716neural networksrobustness learningloss functionadversarial examples |
| spellingShingle | Bingbing Song Ruxin Wang Wei He Wei Zhou Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image Mathematics neural networks robustness learning loss function adversarial examples |
| title | Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image |
| title_full | Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image |
| title_fullStr | Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image |
| title_full_unstemmed | Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image |
| title_short | Robustness Learning via Inference-Softmax Cross Entropy in Misaligned Distribution of Image |
| title_sort | robustness learning via inference softmax cross entropy in misaligned distribution of image |
| topic | neural networks robustness learning loss function adversarial examples |
| url | https://www.mdpi.com/2227-7390/10/19/3716 |
| work_keys_str_mv | AT bingbingsong robustnesslearningviainferencesoftmaxcrossentropyinmisaligneddistributionofimage AT ruxinwang robustnesslearningviainferencesoftmaxcrossentropyinmisaligneddistributionofimage AT weihe robustnesslearningviainferencesoftmaxcrossentropyinmisaligneddistributionofimage AT weizhou robustnesslearningviainferencesoftmaxcrossentropyinmisaligneddistributionofimage |