Early-stage fusion of EEG and fNIRS improves classification of motor imagery
IntroductionMany research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which wer...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-01-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2022.1062889/full |
| _version_ | 1828067970462515200 |
|---|---|
| author | Yang Li Xin Zhang Xin Zhang Dong Ming Dong Ming |
| author_facet | Yang Li Xin Zhang Xin Zhang Dong Ming Dong Ming |
| author_sort | Yang Li |
| collection | DOAJ |
| description | IntroductionMany research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.MethodsIn this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.ResultsThe results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data. |
| first_indexed | 2024-04-10T23:55:17Z |
| format | Article |
| id | doaj.art-1d9dee88923c482e85ef21de954a4db0 |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-04-10T23:55:17Z |
| publishDate | 2023-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-1d9dee88923c482e85ef21de954a4db02023-01-10T13:51:17ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2023-01-011610.3389/fnins.2022.10628891062889Early-stage fusion of EEG and fNIRS improves classification of motor imageryYang Li0Xin Zhang1Xin Zhang2Dong Ming3Dong Ming4Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, ChinaThe Laboratory of Neural Engineering and Rehabilitation, Department of Biomedical Engineering, School of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, ChinaThe Tianjin International Joint Research Center for Neural Engineering, Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, ChinaThe Laboratory of Neural Engineering and Rehabilitation, Department of Biomedical Engineering, School of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, ChinaThe Tianjin International Joint Research Center for Neural Engineering, Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, ChinaIntroductionMany research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.MethodsIn this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.ResultsThe results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.https://www.frontiersin.org/articles/10.3389/fnins.2022.1062889/fullEEGfNIRShybrid-BCImodality fusionmotor imagery |
| spellingShingle | Yang Li Xin Zhang Xin Zhang Dong Ming Dong Ming Early-stage fusion of EEG and fNIRS improves classification of motor imagery Frontiers in Neuroscience EEG fNIRS hybrid-BCI modality fusion motor imagery |
| title | Early-stage fusion of EEG and fNIRS improves classification of motor imagery |
| title_full | Early-stage fusion of EEG and fNIRS improves classification of motor imagery |
| title_fullStr | Early-stage fusion of EEG and fNIRS improves classification of motor imagery |
| title_full_unstemmed | Early-stage fusion of EEG and fNIRS improves classification of motor imagery |
| title_short | Early-stage fusion of EEG and fNIRS improves classification of motor imagery |
| title_sort | early stage fusion of eeg and fnirs improves classification of motor imagery |
| topic | EEG fNIRS hybrid-BCI modality fusion motor imagery |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2022.1062889/full |
| work_keys_str_mv | AT yangli earlystagefusionofeegandfnirsimprovesclassificationofmotorimagery AT xinzhang earlystagefusionofeegandfnirsimprovesclassificationofmotorimagery AT xinzhang earlystagefusionofeegandfnirsimprovesclassificationofmotorimagery AT dongming earlystagefusionofeegandfnirsimprovesclassificationofmotorimagery AT dongming earlystagefusionofeegandfnirsimprovesclassificationofmotorimagery |