Study of mental state recognition using physiological signals in virtual environment
Non-invasive electroencephalogram (EEG) based Brain-Computer interface (BCI) technology enables a new mode of communication and control channel between brain and computer, bypassing the brain’s conventional pathways of nerves and muscles. BCIs can be applied for communication, control and neurofeedb...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis-Master by Research |
| Language: | English |
| Published: |
Nanyang Technological University
2022
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/157151 |
| _version_ | 1811684655733145600 |
|---|---|
| author | Bilgin, Pinar |
| author2 | Guan Cuntai |
| author_facet | Guan Cuntai Bilgin, Pinar |
| author_sort | Bilgin, Pinar |
| collection | NTU |
| description | Non-invasive electroencephalogram (EEG) based Brain-Computer interface (BCI) technology enables a new mode of communication and control channel between brain and computer, bypassing the brain’s conventional pathways of nerves and muscles. BCIs can be applied for communication, control and neurofeedback for healthy people as well as people with neuromuscular disabilities, by identifying mental intentions directly from brain signal modulations, using signal processing and machine learning methods. From the neurofeedback perspective, EEG-based BCI employs involuntary brain signals to gather information about user’s emotional or cognitive states which can further be used in the treatment of patients suffering from mental disorders such as anxiety and depression. Advances via affective computing to recognize, process and simulate emotions which involves both emotion recognition and emotion elicitation, allows neurofeedback-based control of EEG signals that can be used as cost-effective, non-pharmacological treatment options for regulation of mental states. Integrating interdisciplinary fields of Neuroscience, Human-Machine Interaction, and Psychology, it is possible to understand emotional responses of the brain to certain affective stimuli diligently. However, the effectiveness of emotion recognition process heavily depends on two main factors, firstly the quality of acquired data and secondly, recognition algorithms used to classify unique emotions. Quality of the data is greatly affected by the emotion elicitation technique, whereas the common approaches include audio, visual, audio-visual stimuli using non-immersive displays. Recent research shows an increase in immersive displays used for emotion elicitation due to their ability to simulate and evaluate spatial environments under controlled laboratory conditions. In this thesis, an extensive comparative study between non-immersive 2D display and immersive 3D Virtual Reality (VR) display has been presented. To accomplish the same, a cross-over experiment protocol comprising of two distinct emotion eliciting environments relaxation and arousal-presented to the participant either in a 2D monitor or a 3D head-mounted (HMD) VR display have been designed. EEG data is collected during the experiment and analyzed offline to classify emotion elicited by low and high arousal environments using SVM classification of band power features and a couple of deep learning algorithms. The average classification accuracies over subjects are obtained as 66.88% and 59.27% in 3D-VR and 2D-screen group respectively using SVMwhereas it is82.39 % and 72.28%using deep learning algorithms. The performance difference between 3D-VR and 2D screens is statistically significant (p < 0.05), indicating that 3D-VR generates more distinct EEG patterns associated with emotion elicitation. Further, significant differences in band powers from alpha, theta and beta bands are also observed between both groups. For deep learning-based recognition, Convolutional Neural Network (CNN)and Recurrent Neural Network (RNN) have been employed to achieve promising classification of affective states. The deep learning methods can be exploited in future affective BCI studies to contribute further to reveal the mechanisms of the mental state recognition. |
| first_indexed | 2024-10-01T04:32:05Z |
| format | Thesis-Master by Research |
| id | ntu-10356/157151 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T04:32:05Z |
| publishDate | 2022 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1571512024-04-09T02:35:46Z Study of mental state recognition using physiological signals in virtual environment Bilgin, Pinar Guan Cuntai School of Computer Science and Engineering Institute of High Performance Computing, A*STAR Computational Intelligence Lab Kathleen Agres CTGuan@ntu.edu.sg, katagres@nus.edu.sg Engineering::Computer science and engineering Non-invasive electroencephalogram (EEG) based Brain-Computer interface (BCI) technology enables a new mode of communication and control channel between brain and computer, bypassing the brain’s conventional pathways of nerves and muscles. BCIs can be applied for communication, control and neurofeedback for healthy people as well as people with neuromuscular disabilities, by identifying mental intentions directly from brain signal modulations, using signal processing and machine learning methods. From the neurofeedback perspective, EEG-based BCI employs involuntary brain signals to gather information about user’s emotional or cognitive states which can further be used in the treatment of patients suffering from mental disorders such as anxiety and depression. Advances via affective computing to recognize, process and simulate emotions which involves both emotion recognition and emotion elicitation, allows neurofeedback-based control of EEG signals that can be used as cost-effective, non-pharmacological treatment options for regulation of mental states. Integrating interdisciplinary fields of Neuroscience, Human-Machine Interaction, and Psychology, it is possible to understand emotional responses of the brain to certain affective stimuli diligently. However, the effectiveness of emotion recognition process heavily depends on two main factors, firstly the quality of acquired data and secondly, recognition algorithms used to classify unique emotions. Quality of the data is greatly affected by the emotion elicitation technique, whereas the common approaches include audio, visual, audio-visual stimuli using non-immersive displays. Recent research shows an increase in immersive displays used for emotion elicitation due to their ability to simulate and evaluate spatial environments under controlled laboratory conditions. In this thesis, an extensive comparative study between non-immersive 2D display and immersive 3D Virtual Reality (VR) display has been presented. To accomplish the same, a cross-over experiment protocol comprising of two distinct emotion eliciting environments relaxation and arousal-presented to the participant either in a 2D monitor or a 3D head-mounted (HMD) VR display have been designed. EEG data is collected during the experiment and analyzed offline to classify emotion elicited by low and high arousal environments using SVM classification of band power features and a couple of deep learning algorithms. The average classification accuracies over subjects are obtained as 66.88% and 59.27% in 3D-VR and 2D-screen group respectively using SVMwhereas it is82.39 % and 72.28%using deep learning algorithms. The performance difference between 3D-VR and 2D screens is statistically significant (p < 0.05), indicating that 3D-VR generates more distinct EEG patterns associated with emotion elicitation. Further, significant differences in band powers from alpha, theta and beta bands are also observed between both groups. For deep learning-based recognition, Convolutional Neural Network (CNN)and Recurrent Neural Network (RNN) have been employed to achieve promising classification of affective states. The deep learning methods can be exploited in future affective BCI studies to contribute further to reveal the mechanisms of the mental state recognition. Master of Engineering 2022-05-09T12:09:40Z 2022-05-09T12:09:40Z 2021 Thesis-Master by Research Bilgin, P. (2021). Study of mental state recognition using physiological signals in virtual environment. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/157151 https://hdl.handle.net/10356/157151 10.32657/10356/157151 en This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). application/pdf Nanyang Technological University |
| spellingShingle | Engineering::Computer science and engineering Bilgin, Pinar Study of mental state recognition using physiological signals in virtual environment |
| title | Study of mental state recognition using physiological signals in virtual environment |
| title_full | Study of mental state recognition using physiological signals in virtual environment |
| title_fullStr | Study of mental state recognition using physiological signals in virtual environment |
| title_full_unstemmed | Study of mental state recognition using physiological signals in virtual environment |
| title_short | Study of mental state recognition using physiological signals in virtual environment |
| title_sort | study of mental state recognition using physiological signals in virtual environment |
| topic | Engineering::Computer science and engineering |
| url | https://hdl.handle.net/10356/157151 |
| work_keys_str_mv | AT bilginpinar studyofmentalstaterecognitionusingphysiologicalsignalsinvirtualenvironment |