FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network
Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin te...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-09-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/20/18/5328 |
| _version_ | 1797553378959556608 |
|---|---|
| author | Clarence Tan Gerardo Ceballos Nikola Kasabov Narayan Puthanmadam Subramaniyam |
| author_facet | Clarence Tan Gerardo Ceballos Nikola Kasabov Narayan Puthanmadam Subramaniyam |
| author_sort | Clarence Tan |
| collection | DOAJ |
| description | Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem. |
| first_indexed | 2024-03-10T16:15:36Z |
| format | Article |
| id | doaj.art-5ac6f8aa49f34062b7fc18763d172625 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T16:15:36Z |
| publishDate | 2020-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-5ac6f8aa49f34062b7fc18763d1726252023-11-20T14:06:44ZengMDPI AGSensors1424-82202020-09-012018532810.3390/s20185328FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural NetworkClarence Tan0Gerardo Ceballos1Nikola Kasabov2Narayan Puthanmadam Subramaniyam3Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, Auckland 1010, New ZealandSchool of Electrical Engineering, University of Los Andes, Merida 5101, VenezuelaKnowledge Engineering and Discovery Research Institute, Auckland University of Technology, Auckland 1010, New ZealandFaculty of Medicine and Health Technology and BioMediTech Institute, Tampere University, 33520 Tampere, FinlandUsing multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.https://www.mdpi.com/1424-8220/20/18/5328facial emotion recognitionEvolving Spiking Neural Networks (eSNNs)Spatio-temporal dataNeuCubemultimodal data |
| spellingShingle | Clarence Tan Gerardo Ceballos Nikola Kasabov Narayan Puthanmadam Subramaniyam FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network Sensors facial emotion recognition Evolving Spiking Neural Networks (eSNNs) Spatio-temporal data NeuCube multimodal data |
| title | FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network |
| title_full | FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network |
| title_fullStr | FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network |
| title_full_unstemmed | FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network |
| title_short | FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network |
| title_sort | fusionsense emotion classification using feature fusion of multimodal data and deep learning in a brain inspired spiking neural network |
| topic | facial emotion recognition Evolving Spiking Neural Networks (eSNNs) Spatio-temporal data NeuCube multimodal data |
| url | https://www.mdpi.com/1424-8220/20/18/5328 |
| work_keys_str_mv | AT clarencetan fusionsenseemotionclassificationusingfeaturefusionofmultimodaldataanddeeplearninginabraininspiredspikingneuralnetwork AT gerardoceballos fusionsenseemotionclassificationusingfeaturefusionofmultimodaldataanddeeplearninginabraininspiredspikingneuralnetwork AT nikolakasabov fusionsenseemotionclassificationusingfeaturefusionofmultimodaldataanddeeplearninginabraininspiredspikingneuralnetwork AT narayanputhanmadamsubramaniyam fusionsenseemotionclassificationusingfeaturefusionofmultimodaldataanddeeplearninginabraininspiredspikingneuralnetwork |