Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition
Emotion recognition is a key attribute for realizing advances in human–computer interaction, especially when using non-intrusive physiological sensors, such as electroencephalograph (EEG) and electrocardiograph. Although functional connectivity of EEG has been utilized for emotion recognition, the g...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/21/8198 |
| _version_ | 1797466535414988800 |
|---|---|
| author | Vasileios-Rafail Xefteris Athina Tsanousa Nefeli Georgakopoulou Sotiris Diplaris Stefanos Vrochidis Ioannis Kompatsiaris |
| author_facet | Vasileios-Rafail Xefteris Athina Tsanousa Nefeli Georgakopoulou Sotiris Diplaris Stefanos Vrochidis Ioannis Kompatsiaris |
| author_sort | Vasileios-Rafail Xefteris |
| collection | DOAJ |
| description | Emotion recognition is a key attribute for realizing advances in human–computer interaction, especially when using non-intrusive physiological sensors, such as electroencephalograph (EEG) and electrocardiograph. Although functional connectivity of EEG has been utilized for emotion recognition, the graph theory analysis of EEG connectivity patterns has not been adequately explored. The exploitation of brain network characteristics could provide valuable information regarding emotions, while the combination of EEG and peripheral physiological signals can reveal correlation patterns of human internal state. In this work, a graph theoretical analysis of EEG functional connectivity patterns along with fusion between EEG and peripheral physiological signals for emotion recognition has been proposed. After extracting functional connectivity from EEG signals, both global and local graph theory features are extracted. Those features are concatenated with statistical features from peripheral physiological signals and fed to different classifiers and a Convolutional Neural Network (CNN) for emotion recognition. The average accuracy on the DEAP dataset using CNN was 55.62% and 57.38% for subject-independent valence and arousal classification, respectively, and 83.94% and 83.87% for subject-dependent classification. Those scores went up to 75.44% and 78.77% for subject-independent classification and 88.27% and 90.84% for subject-dependent classification using a feature selection algorithm, exceeding the current state-of-the-art results. |
| first_indexed | 2024-03-09T18:41:10Z |
| format | Article |
| id | doaj.art-859dbc0af23b48cc8dd1e89497ef8827 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T18:41:10Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-859dbc0af23b48cc8dd1e89497ef88272023-11-24T06:44:15ZengMDPI AGSensors1424-82202022-10-012221819810.3390/s22218198Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion RecognitionVasileios-Rafail Xefteris0Athina Tsanousa1Nefeli Georgakopoulou2Sotiris Diplaris3Stefanos Vrochidis4Ioannis Kompatsiaris5Centre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceCentre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceCentre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceCentre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceCentre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceCentre for Research and Technology Hellas, Information Technologies Institute, 6th Km Charilaou-Thermi, 57001 Thessaloniki, GreeceEmotion recognition is a key attribute for realizing advances in human–computer interaction, especially when using non-intrusive physiological sensors, such as electroencephalograph (EEG) and electrocardiograph. Although functional connectivity of EEG has been utilized for emotion recognition, the graph theory analysis of EEG connectivity patterns has not been adequately explored. The exploitation of brain network characteristics could provide valuable information regarding emotions, while the combination of EEG and peripheral physiological signals can reveal correlation patterns of human internal state. In this work, a graph theoretical analysis of EEG functional connectivity patterns along with fusion between EEG and peripheral physiological signals for emotion recognition has been proposed. After extracting functional connectivity from EEG signals, both global and local graph theory features are extracted. Those features are concatenated with statistical features from peripheral physiological signals and fed to different classifiers and a Convolutional Neural Network (CNN) for emotion recognition. The average accuracy on the DEAP dataset using CNN was 55.62% and 57.38% for subject-independent valence and arousal classification, respectively, and 83.94% and 83.87% for subject-dependent classification. Those scores went up to 75.44% and 78.77% for subject-independent classification and 88.27% and 90.84% for subject-dependent classification using a feature selection algorithm, exceeding the current state-of-the-art results.https://www.mdpi.com/1424-8220/22/21/8198emotion recognitionEEGmultimodal physiological signalsfunctional connectivitygraph theorymultimodal fusion |
| spellingShingle | Vasileios-Rafail Xefteris Athina Tsanousa Nefeli Georgakopoulou Sotiris Diplaris Stefanos Vrochidis Ioannis Kompatsiaris Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition Sensors emotion recognition EEG multimodal physiological signals functional connectivity graph theory multimodal fusion |
| title | Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition |
| title_full | Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition |
| title_fullStr | Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition |
| title_full_unstemmed | Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition |
| title_short | Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition |
| title_sort | graph theoretical analysis of eeg functional connectivity patterns and fusion with physiological signals for emotion recognition |
| topic | emotion recognition EEG multimodal physiological signals functional connectivity graph theory multimodal fusion |
| url | https://www.mdpi.com/1424-8220/22/21/8198 |
| work_keys_str_mv | AT vasileiosrafailxefteris graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition AT athinatsanousa graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition AT nefeligeorgakopoulou graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition AT sotirisdiplaris graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition AT stefanosvrochidis graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition AT ioanniskompatsiaris graphtheoreticalanalysisofeegfunctionalconnectivitypatternsandfusionwithphysiologicalsignalsforemotionrecognition |