EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition
Fine-grained recognition has many applications in many fields and aims to identify targets from subcategories. This is a highly challenging task due to the minor differences between subcategories. Both modal missing and adversarial sample attacks are easily encountered in fine-grained recognition ta...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/3/1640 |
| _version_ | 1797625092464705536 |
|---|---|
| author | Rongping Zou Bin Zhu Yi Chen Bo Xie Bin Shao |
| author_facet | Rongping Zou Bin Zhu Yi Chen Bo Xie Bin Shao |
| author_sort | Rongping Zou |
| collection | DOAJ |
| description | Fine-grained recognition has many applications in many fields and aims to identify targets from subcategories. This is a highly challenging task due to the minor differences between subcategories. Both modal missing and adversarial sample attacks are easily encountered in fine-grained recognition tasks based on multimodal data. These situations can easily lead to the model needing to be fixed. An Enhanced Framework for the Complementarity of Multimodal Features (EFCMF) is proposed in this study to solve this problem. The model’s learning of multimodal data complementarity is enhanced by randomly deactivating modal features in the constructed multimodal fine-grained recognition model. The results show that the model gains the ability to handle modal missing without additional training of the model and can achieve 91.14% and 99.31% accuracy on Birds and Flowers datasets. The average accuracy of EFCMF on the two datasets is 52.85%, which is 27.13% higher than that of Bi-modal PMA when facing four adversarial example attacks, namely FGSM, BIM, PGD and C&W. In the face of missing modal cases, the average accuracy of EFCMF is 76.33% on both datasets respectively, which is 32.63% higher than that of Bi-modal PMA. Compared with existing methods, EFCMF is robust in the face of modal missing and adversarial example attacks in multimodal fine-grained recognition tasks. The source code is available at https://github.com/RPZ97/EFCMF (accessed on 8 January 2023). |
| first_indexed | 2024-03-11T09:51:56Z |
| format | Article |
| id | doaj.art-53e89a6621d14df194301c78bb49385a |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T09:51:56Z |
| publishDate | 2023-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-53e89a6621d14df194301c78bb49385a2023-11-16T16:08:23ZengMDPI AGApplied Sciences2076-34172023-01-01133164010.3390/app13031640EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained RecognitionRongping Zou0Bin Zhu1Yi Chen2Bo Xie3Bin Shao4College of Electronic Engineering, National University of Defense Technology, Hefei 230037, ChinaCollege of Electronic Engineering, National University of Defense Technology, Hefei 230037, ChinaCollege of Electronic Engineering, National University of Defense Technology, Hefei 230037, ChinaCollege of Electronic Engineering, National University of Defense Technology, Hefei 230037, ChinaCollege of Electronic Engineering, National University of Defense Technology, Hefei 230037, ChinaFine-grained recognition has many applications in many fields and aims to identify targets from subcategories. This is a highly challenging task due to the minor differences between subcategories. Both modal missing and adversarial sample attacks are easily encountered in fine-grained recognition tasks based on multimodal data. These situations can easily lead to the model needing to be fixed. An Enhanced Framework for the Complementarity of Multimodal Features (EFCMF) is proposed in this study to solve this problem. The model’s learning of multimodal data complementarity is enhanced by randomly deactivating modal features in the constructed multimodal fine-grained recognition model. The results show that the model gains the ability to handle modal missing without additional training of the model and can achieve 91.14% and 99.31% accuracy on Birds and Flowers datasets. The average accuracy of EFCMF on the two datasets is 52.85%, which is 27.13% higher than that of Bi-modal PMA when facing four adversarial example attacks, namely FGSM, BIM, PGD and C&W. In the face of missing modal cases, the average accuracy of EFCMF is 76.33% on both datasets respectively, which is 32.63% higher than that of Bi-modal PMA. Compared with existing methods, EFCMF is robust in the face of modal missing and adversarial example attacks in multimodal fine-grained recognition tasks. The source code is available at https://github.com/RPZ97/EFCMF (accessed on 8 January 2023).https://www.mdpi.com/2076-3417/13/3/1640fine-grained recognitionmultimodalmodal missingadversarial examples |
| spellingShingle | Rongping Zou Bin Zhu Yi Chen Bo Xie Bin Shao EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition Applied Sciences fine-grained recognition multimodal modal missing adversarial examples |
| title | EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition |
| title_full | EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition |
| title_fullStr | EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition |
| title_full_unstemmed | EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition |
| title_short | EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition |
| title_sort | efcmf a multimodal robustness enhancement framework for fine grained recognition |
| topic | fine-grained recognition multimodal modal missing adversarial examples |
| url | https://www.mdpi.com/2076-3417/13/3/1640 |
| work_keys_str_mv | AT rongpingzou efcmfamultimodalrobustnessenhancementframeworkforfinegrainedrecognition AT binzhu efcmfamultimodalrobustnessenhancementframeworkforfinegrainedrecognition AT yichen efcmfamultimodalrobustnessenhancementframeworkforfinegrainedrecognition AT boxie efcmfamultimodalrobustnessenhancementframeworkforfinegrainedrecognition AT binshao efcmfamultimodalrobustnessenhancementframeworkforfinegrainedrecognition |