An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals
Emotion charting using multimodal signals has gained great demand for stroke-affected patients, for psychiatrists while examining patients, and for neuromarketing applications. Multimodal signals for emotion charting include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and ga...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/23/9480 |
| _version_ | 1797462096642834432 |
|---|---|
| author | Amna Waheed Awan Syed Muhammad Usman Shehzad Khalid Aamir Anwar Roobaea Alroobaea Saddam Hussain Jasem Almotiri Syed Sajid Ullah Muhammad Usman Akram |
| author_facet | Amna Waheed Awan Syed Muhammad Usman Shehzad Khalid Aamir Anwar Roobaea Alroobaea Saddam Hussain Jasem Almotiri Syed Sajid Ullah Muhammad Usman Akram |
| author_sort | Amna Waheed Awan |
| collection | DOAJ |
| description | Emotion charting using multimodal signals has gained great demand for stroke-affected patients, for psychiatrists while examining patients, and for neuromarketing applications. Multimodal signals for emotion charting include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and galvanic skin response (GSR) signals. EEG, ECG, and GSR are also known as physiological signals, which can be used for identification of human emotions. Due to the unbiased nature of physiological signals, this field has become a great motivation in recent research as physiological signals are generated autonomously from human central nervous system. Researchers have developed multiple methods for the classification of these signals for emotion detection. However, due to the non-linear nature of these signals and the inclusion of noise, while recording, accurate classification of physiological signals is a challenge for emotion charting. Valence and arousal are two important states for emotion detection; therefore, this paper presents a novel ensemble learning method based on deep learning for the classification of four different emotional states including high valence and high arousal (HVHA), low valence and low arousal (LVLA), high valence and low arousal (HVLA) and low valence high arousal (LVHA). In the proposed method, multimodal signals (EEG, ECG, and GSR) are preprocessed using bandpass filtering and independent components analysis (ICA) for noise removal in EEG signals followed by discrete wavelet transform for time domain to frequency domain conversion. Discrete wavelet transform results in spectrograms of the physiological signal and then features are extracted using stacked autoencoders from those spectrograms. A feature vector is obtained from the bottleneck layer of the autoencoder and is fed to three classifiers SVM (support vector machine), RF (random forest), and LSTM (long short-term memory) followed by majority voting as ensemble classification. The proposed system is trained and tested on the AMIGOS dataset with <i>k</i>-fold cross-validation. The proposed system obtained the highest accuracy of 94.5% and shows improved results of the proposed method compared with other state-of-the-art methods. |
| first_indexed | 2024-03-09T17:31:38Z |
| format | Article |
| id | doaj.art-1bb3e19c11274dceb9c52fb6c38eadcb |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T17:31:38Z |
| publishDate | 2022-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-1bb3e19c11274dceb9c52fb6c38eadcb2023-11-24T12:15:05ZengMDPI AGSensors1424-82202022-12-012223948010.3390/s22239480An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological SignalsAmna Waheed Awan0Syed Muhammad Usman1Shehzad Khalid2Aamir Anwar3Roobaea Alroobaea4Saddam Hussain5Jasem Almotiri6Syed Sajid Ullah7Muhammad Usman Akram8Department of Computer Engineering, Bahria University, Islamabad 44000, PakistanDepartment of Creative Technologies, Faculty of Computing and AI, Air University, Islamabad 44000, PakistanDepartment of Computer Engineering, Bahria University, Islamabad 44000, PakistanSchool of Computing and Engineering, The University of West London, London W5 5RF, UKDepartment of Computer Science, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi ArabiaSchool of Digital Science, Universiti Brunei Darussalam, Jalan Tungku Link, Gadong BE1410, BruneiDepartment of Computer Science, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi ArabiaDepartment of Information and Communication Technology, University of Agder (UiA), N-4898 Grimstad, NorwayCollege of Eletrical and Mechanical Engineering (E & ME), National University of Science and Technology (NUST), Islamabad 44000, PakistanEmotion charting using multimodal signals has gained great demand for stroke-affected patients, for psychiatrists while examining patients, and for neuromarketing applications. Multimodal signals for emotion charting include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and galvanic skin response (GSR) signals. EEG, ECG, and GSR are also known as physiological signals, which can be used for identification of human emotions. Due to the unbiased nature of physiological signals, this field has become a great motivation in recent research as physiological signals are generated autonomously from human central nervous system. Researchers have developed multiple methods for the classification of these signals for emotion detection. However, due to the non-linear nature of these signals and the inclusion of noise, while recording, accurate classification of physiological signals is a challenge for emotion charting. Valence and arousal are two important states for emotion detection; therefore, this paper presents a novel ensemble learning method based on deep learning for the classification of four different emotional states including high valence and high arousal (HVHA), low valence and low arousal (LVLA), high valence and low arousal (HVLA) and low valence high arousal (LVHA). In the proposed method, multimodal signals (EEG, ECG, and GSR) are preprocessed using bandpass filtering and independent components analysis (ICA) for noise removal in EEG signals followed by discrete wavelet transform for time domain to frequency domain conversion. Discrete wavelet transform results in spectrograms of the physiological signal and then features are extracted using stacked autoencoders from those spectrograms. A feature vector is obtained from the bottleneck layer of the autoencoder and is fed to three classifiers SVM (support vector machine), RF (random forest), and LSTM (long short-term memory) followed by majority voting as ensemble classification. The proposed system is trained and tested on the AMIGOS dataset with <i>k</i>-fold cross-validation. The proposed system obtained the highest accuracy of 94.5% and shows improved results of the proposed method compared with other state-of-the-art methods.https://www.mdpi.com/1424-8220/22/23/9480emotion chartingEEG signalsphysiological signalsECG signalsICAstacked autoencoder |
| spellingShingle | Amna Waheed Awan Syed Muhammad Usman Shehzad Khalid Aamir Anwar Roobaea Alroobaea Saddam Hussain Jasem Almotiri Syed Sajid Ullah Muhammad Usman Akram An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals Sensors emotion charting EEG signals physiological signals ECG signals ICA stacked autoencoder |
| title | An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals |
| title_full | An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals |
| title_fullStr | An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals |
| title_full_unstemmed | An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals |
| title_short | An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals |
| title_sort | ensemble learning method for emotion charting using multimodal physiological signals |
| topic | emotion charting EEG signals physiological signals ECG signals ICA stacked autoencoder |
| url | https://www.mdpi.com/1424-8220/22/23/9480 |
| work_keys_str_mv | AT amnawaheedawan anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT syedmuhammadusman anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT shehzadkhalid anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT aamiranwar anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT roobaeaalroobaea anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT saddamhussain anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT jasemalmotiri anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT syedsajidullah anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT muhammadusmanakram anensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT amnawaheedawan ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT syedmuhammadusman ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT shehzadkhalid ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT aamiranwar ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT roobaeaalroobaea ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT saddamhussain ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT jasemalmotiri ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT syedsajidullah ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals AT muhammadusmanakram ensemblelearningmethodforemotionchartingusingmultimodalphysiologicalsignals |