AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network
Micro-expressions are the small, brief facial expression changes that humans momentarily show during emotional experiences, and their data annotation is complicated, which leads to the scarcity of micro-expression data. To extract salient and distinguishing features from a limited dataset, we propos...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/25/7/1064 |
| _version_ | 1797589363528302592 |
|---|---|
| author | Chenghao Fu Wenzhong Yang Danny Chen Fuyuan Wei |
| author_facet | Chenghao Fu Wenzhong Yang Danny Chen Fuyuan Wei |
| author_sort | Chenghao Fu |
| collection | DOAJ |
| description | Micro-expressions are the small, brief facial expression changes that humans momentarily show during emotional experiences, and their data annotation is complicated, which leads to the scarcity of micro-expression data. To extract salient and distinguishing features from a limited dataset, we propose an attention-based multi-scale, multi-modal, multi-branch flow network to thoroughly learn the motion information of micro-expressions by exploiting the attention mechanism and the complementary properties between different optical flow information. First, we extract optical flow information (horizontal optical flow, vertical optical flow, and optical strain) based on the onset and apex frames of micro-expression videos, and each branch learns one kind of optical flow information separately. Second, we propose a multi-scale fusion module to extract more prosperous and more stable feature expressions using spatial attention to focus on locally important information at each scale. Then, we design a multi-optical flow feature reweighting module to adaptively select features for each optical flow separately by channel attention. Finally, to better integrate the information of the three branches and to alleviate the problem of uneven distribution of micro-expression samples, we introduce a logarithmically adjusted prior knowledge weighting loss. This loss function weights the prediction scores of samples from different categories to mitigate the negative impact of category imbalance during the classification process. The effectiveness of the proposed model is demonstrated through extensive experiments and feature visualization on three benchmark datasets (CASMEII, SAMM, and SMIC), and its performance is comparable to that of state-of-the-art methods. |
| first_indexed | 2024-03-11T01:05:34Z |
| format | Article |
| id | doaj.art-46ffb9b6940a45409994dfda837ffe97 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-11T01:05:34Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-46ffb9b6940a45409994dfda837ffe972023-11-18T19:14:15ZengMDPI AGEntropy1099-43002023-07-01257106410.3390/e25071064AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow NetworkChenghao Fu0Wenzhong Yang1Danny Chen2Fuyuan Wei3School of Information Science and Engineering, Xinjiang University, Urumqi 830017, ChinaSchool of Information Science and Engineering, Xinjiang University, Urumqi 830017, ChinaSchool of Information Science and Engineering, Xinjiang University, Urumqi 830017, ChinaSchool of Information Science and Engineering, Xinjiang University, Urumqi 830017, ChinaMicro-expressions are the small, brief facial expression changes that humans momentarily show during emotional experiences, and their data annotation is complicated, which leads to the scarcity of micro-expression data. To extract salient and distinguishing features from a limited dataset, we propose an attention-based multi-scale, multi-modal, multi-branch flow network to thoroughly learn the motion information of micro-expressions by exploiting the attention mechanism and the complementary properties between different optical flow information. First, we extract optical flow information (horizontal optical flow, vertical optical flow, and optical strain) based on the onset and apex frames of micro-expression videos, and each branch learns one kind of optical flow information separately. Second, we propose a multi-scale fusion module to extract more prosperous and more stable feature expressions using spatial attention to focus on locally important information at each scale. Then, we design a multi-optical flow feature reweighting module to adaptively select features for each optical flow separately by channel attention. Finally, to better integrate the information of the three branches and to alleviate the problem of uneven distribution of micro-expression samples, we introduce a logarithmically adjusted prior knowledge weighting loss. This loss function weights the prediction scores of samples from different categories to mitigate the negative impact of category imbalance during the classification process. The effectiveness of the proposed model is demonstrated through extensive experiments and feature visualization on three benchmark datasets (CASMEII, SAMM, and SMIC), and its performance is comparable to that of state-of-the-art methods.https://www.mdpi.com/1099-4300/25/7/1064micro-expression recognitionattention mechanismslogit-adjusted loss |
| spellingShingle | Chenghao Fu Wenzhong Yang Danny Chen Fuyuan Wei AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network Entropy micro-expression recognition attention mechanisms logit-adjusted loss |
| title | AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network |
| title_full | AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network |
| title_fullStr | AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network |
| title_full_unstemmed | AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network |
| title_short | AM3F-FlowNet: Attention-Based Multi-Scale Multi-Branch Flow Network |
| title_sort | am3f flownet attention based multi scale multi branch flow network |
| topic | micro-expression recognition attention mechanisms logit-adjusted loss |
| url | https://www.mdpi.com/1099-4300/25/7/1064 |
| work_keys_str_mv | AT chenghaofu am3fflownetattentionbasedmultiscalemultibranchflownetwork AT wenzhongyang am3fflownetattentionbasedmultiscalemultibranchflownetwork AT dannychen am3fflownetattentionbasedmultiscalemultibranchflownetwork AT fuyuanwei am3fflownetattentionbasedmultiscalemultibranchflownetwork |