Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition
Background: Recognizing human emotions by micro-expression recognition is one of the most critical issues in human-computer interaction applications. Cross-database micro-expression recognition (CDMER) is an increasingly significant problem in micro-expression recognition and analysis in recent year...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co., Ltd.
2023-04-01
|
| Series: | Virtual Reality & Intelligent Hardware |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2096579622000316 |
| _version_ | 1797832784427876352 |
|---|---|
| author | Yuhan Ran Wenming ZHENG Yuan ZONG Jiateng LIU |
| author_facet | Yuhan Ran Wenming ZHENG Yuan ZONG Jiateng LIU |
| author_sort | Yuhan Ran |
| collection | DOAJ |
| description | Background: Recognizing human emotions by micro-expression recognition is one of the most critical issues in human-computer interaction applications. Cross-database micro-expression recognition (CDMER) is an increasingly significant problem in micro-expression recognition and analysis in recent years. Since the training data and testing data in CDMER come from different micro-expression databases, CDMER is more challenging than the conventional micro-expression recognition. Methods: In this paper, an Adaptive Spatio-Temporal Attention Neural Network (ASTANN) using attention mechanism is presented to deal with this challenging and critical problem. To this end, the micro-expression database SMIC and CASME II are firstly preprocessed by optical flow approach, which extract motion information among video frames that represents discriminative features of micro-expression. After preprocessing, a novel adaptive framework with spatio-temporal attention module is designed to assign spatial and temporal weights to enhance the most discriminative features. Then the deep neural network extracts cross-domain feature, in which the second-order statistics of the sample features in source domain is aligned with the ones in the target domain by minimizing the correlation alignment (CORAL) loss such that source and target database share similar distributions. Results: To evaluate the performance of ASTANN, experiments are conducted based on SMIC and CASME II databases under a standard experimental evaluation protocol of CDMER. The experimental results demonstrate that ASTANN outperforms other methods in relevant cross-database tasks. Conclusions: Extensive experiments are conducted on the benchmark tasks and results show that ASTANN achieves the best performance over all other approaches. This remarkable performance demonstrates the superiority of our method for solving CDMER problem. |
| first_indexed | 2024-04-09T14:13:22Z |
| format | Article |
| id | doaj.art-a95d8dc981e044f3ade2de4a8d1ba072 |
| institution | Directory Open Access Journal |
| issn | 2096-5796 |
| language | English |
| last_indexed | 2024-04-09T14:13:22Z |
| publishDate | 2023-04-01 |
| publisher | KeAi Communications Co., Ltd. |
| record_format | Article |
| series | Virtual Reality & Intelligent Hardware |
| spelling | doaj.art-a95d8dc981e044f3ade2de4a8d1ba0722023-05-06T04:37:34ZengKeAi Communications Co., Ltd.Virtual Reality & Intelligent Hardware2096-57962023-04-0152142156Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognitionYuhan Ran0Wenming ZHENG1Yuan ZONG2Jiateng LIU3Key Laboratory of Child Development and Learning Science of Ministry of Education, Research Center for Learning Science, Southeast University, Nanjing 210096, China; School of Cyber Science and Engineering, Southeast University, Nanjing 210096, ChinaKey Laboratory of Child Development and Learning Science of Ministry of Education, Research Center for Learning Science, Southeast University, Nanjing 210096, China; School of Biological Science & Medical Engineering, Southeast University, Nanjing 210096, China; Corresponding author. wenmingKey Laboratory of Child Development and Learning Science of Ministry of Education, Research Center for Learning Science, Southeast University, Nanjing 210096, China; School of Biological Science & Medical Engineering, Southeast University, Nanjing 210096, ChinaKey Laboratory of Child Development and Learning Science of Ministry of Education, Research Center for Learning Science, Southeast University, Nanjing 210096, China; School of Biological Science & Medical Engineering, Southeast University, Nanjing 210096, ChinaBackground: Recognizing human emotions by micro-expression recognition is one of the most critical issues in human-computer interaction applications. Cross-database micro-expression recognition (CDMER) is an increasingly significant problem in micro-expression recognition and analysis in recent years. Since the training data and testing data in CDMER come from different micro-expression databases, CDMER is more challenging than the conventional micro-expression recognition. Methods: In this paper, an Adaptive Spatio-Temporal Attention Neural Network (ASTANN) using attention mechanism is presented to deal with this challenging and critical problem. To this end, the micro-expression database SMIC and CASME II are firstly preprocessed by optical flow approach, which extract motion information among video frames that represents discriminative features of micro-expression. After preprocessing, a novel adaptive framework with spatio-temporal attention module is designed to assign spatial and temporal weights to enhance the most discriminative features. Then the deep neural network extracts cross-domain feature, in which the second-order statistics of the sample features in source domain is aligned with the ones in the target domain by minimizing the correlation alignment (CORAL) loss such that source and target database share similar distributions. Results: To evaluate the performance of ASTANN, experiments are conducted based on SMIC and CASME II databases under a standard experimental evaluation protocol of CDMER. The experimental results demonstrate that ASTANN outperforms other methods in relevant cross-database tasks. Conclusions: Extensive experiments are conducted on the benchmark tasks and results show that ASTANN achieves the best performance over all other approaches. This remarkable performance demonstrates the superiority of our method for solving CDMER problem.http://www.sciencedirect.com/science/article/pii/S2096579622000316Cross-database micro-expression recognitionDeep learningAttention mechanismDomain adaption |
| spellingShingle | Yuhan Ran Wenming ZHENG Yuan ZONG Jiateng LIU Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition Virtual Reality & Intelligent Hardware Cross-database micro-expression recognition Deep learning Attention mechanism Domain adaption |
| title | Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition |
| title_full | Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition |
| title_fullStr | Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition |
| title_full_unstemmed | Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition |
| title_short | Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition |
| title_sort | adaptive spatio temporal attention neural network for crossdatabase micro expression recognition |
| topic | Cross-database micro-expression recognition Deep learning Attention mechanism Domain adaption |
| url | http://www.sciencedirect.com/science/article/pii/S2096579622000316 |
| work_keys_str_mv | AT yuhanran adaptivespatiotemporalattentionneuralnetworkforcrossdatabasemicroexpressionrecognition AT wenmingzheng adaptivespatiotemporalattentionneuralnetworkforcrossdatabasemicroexpressionrecognition AT yuanzong adaptivespatiotemporalattentionneuralnetworkforcrossdatabasemicroexpressionrecognition AT jiatengliu adaptivespatiotemporalattentionneuralnetworkforcrossdatabasemicroexpressionrecognition |