An improved multi-input deep convolutional neural network for automatic emotion recognition
Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-10-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2022.965871/full |
| _version_ | 1811181377336377344 |
|---|---|
| author | Peiji Chen Bochao Zou Abdelkader Nasreddine Belkacem Xiangwen Lyu Xixi Zhao Weibo Yi Zhaoyang Huang Jun Liang Chao Chen Chao Chen |
| author_facet | Peiji Chen Bochao Zou Abdelkader Nasreddine Belkacem Xiangwen Lyu Xixi Zhao Weibo Yi Zhaoyang Huang Jun Liang Chao Chen Chao Chen |
| author_sort | Peiji Chen |
| collection | DOAJ |
| description | Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals. |
| first_indexed | 2024-04-11T09:15:40Z |
| format | Article |
| id | doaj.art-1d1e1d85277c47178512f4844acd2f79 |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-04-11T09:15:40Z |
| publishDate | 2022-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-1d1e1d85277c47178512f4844acd2f792022-12-22T04:32:20ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2022-10-011610.3389/fnins.2022.965871965871An improved multi-input deep convolutional neural network for automatic emotion recognitionPeiji Chen0Bochao Zou1Abdelkader Nasreddine Belkacem2Xiangwen Lyu3Xixi Zhao4Weibo Yi5Zhaoyang Huang6Jun Liang7Chao Chen8Chao Chen9Key Laboratory of Complex System Control Theory and Application, Tianjin University of Technology, Tianjin, ChinaSchool of Computer and Communication Engineering, University of Science and Technology, Beijing, ChinaDepartment of Computer and Network Engineering, College of Information Technology, United Arab Emirates University (UAEU), Al Ain, United Arab EmiratesNational Engineering Laboratory for Risk Perception and Prevention, Beijing, ChinaBeijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Capital Medical University, Beijing, ChinaBeijing Machine and Equipment Institute, Beijing, ChinaDepartment of Neurology, Xuanwu Hospital, Capital Medical University, Beijing, ChinaDepartment of Rehabilitation, Tianjin Medical University General Hospital, Tianjin, ChinaKey Laboratory of Complex System Control Theory and Application, Tianjin University of Technology, Tianjin, ChinaAcademy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, ChinaCurrent decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.https://www.frontiersin.org/articles/10.3389/fnins.2022.965871/fullbiological signalsmulti-modalityemotion recognitionconvolutional neural networkmachine learning |
| spellingShingle | Peiji Chen Bochao Zou Abdelkader Nasreddine Belkacem Xiangwen Lyu Xixi Zhao Weibo Yi Zhaoyang Huang Jun Liang Chao Chen Chao Chen An improved multi-input deep convolutional neural network for automatic emotion recognition Frontiers in Neuroscience biological signals multi-modality emotion recognition convolutional neural network machine learning |
| title | An improved multi-input deep convolutional neural network for automatic emotion recognition |
| title_full | An improved multi-input deep convolutional neural network for automatic emotion recognition |
| title_fullStr | An improved multi-input deep convolutional neural network for automatic emotion recognition |
| title_full_unstemmed | An improved multi-input deep convolutional neural network for automatic emotion recognition |
| title_short | An improved multi-input deep convolutional neural network for automatic emotion recognition |
| title_sort | improved multi input deep convolutional neural network for automatic emotion recognition |
| topic | biological signals multi-modality emotion recognition convolutional neural network machine learning |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2022.965871/full |
| work_keys_str_mv | AT peijichen animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT bochaozou animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT abdelkadernasreddinebelkacem animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT xiangwenlyu animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT xixizhao animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT weiboyi animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT zhaoyanghuang animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT junliang animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT chaochen animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT chaochen animprovedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT peijichen improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT bochaozou improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT abdelkadernasreddinebelkacem improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT xiangwenlyu improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT xixizhao improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT weiboyi improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT zhaoyanghuang improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT junliang improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT chaochen improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition AT chaochen improvedmultiinputdeepconvolutionalneuralnetworkforautomaticemotionrecognition |