A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment
At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and ey...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-03-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1146415/full |
| _version_ | 1797859699200098304 |
|---|---|
| author | Rongxiao Guo Yanfei Lin Xi Luo Xiaorong Gao Shangen Zhang |
| author_facet | Rongxiao Guo Yanfei Lin Xi Luo Xiaorong Gao Shangen Zhang |
| author_sort | Rongxiao Guo |
| collection | DOAJ |
| description | At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system. |
| first_indexed | 2024-04-09T21:34:50Z |
| format | Article |
| id | doaj.art-7db92b3169d142b2aed07852de767ea5 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-04-09T21:34:50Z |
| publishDate | 2023-03-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-7db92b3169d142b2aed07852de767ea52023-03-27T05:01:57ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182023-03-011710.3389/fnbot.2023.11464151146415A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environmentRongxiao Guo0Yanfei Lin1Xi Luo2Xiaorong Gao3Shangen Zhang4School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, ChinaSchool of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, ChinaSchool of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, ChinaDepartment of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, ChinaSchool of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, ChinaAt present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.https://www.frontiersin.org/articles/10.3389/fnbot.2023.1146415/fullsteady-state visual evoked potentials (SSVEP)eye-trackinghybrid brain-computer interface (h-BCI)robotic armvirtual reality (VR) |
| spellingShingle | Rongxiao Guo Yanfei Lin Xi Luo Xiaorong Gao Shangen Zhang A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment Frontiers in Neurorobotics steady-state visual evoked potentials (SSVEP) eye-tracking hybrid brain-computer interface (h-BCI) robotic arm virtual reality (VR) |
| title | A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment |
| title_full | A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment |
| title_fullStr | A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment |
| title_full_unstemmed | A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment |
| title_short | A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment |
| title_sort | robotic arm control system with simultaneous and sequential modes combining eye tracking with steady state visual evoked potential in virtual reality environment |
| topic | steady-state visual evoked potentials (SSVEP) eye-tracking hybrid brain-computer interface (h-BCI) robotic arm virtual reality (VR) |
| url | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1146415/full |
| work_keys_str_mv | AT rongxiaoguo aroboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT yanfeilin aroboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT xiluo aroboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT xiaoronggao aroboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT shangenzhang aroboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT rongxiaoguo roboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT yanfeilin roboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT xiluo roboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT xiaoronggao roboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment AT shangenzhang roboticarmcontrolsystemwithsimultaneousandsequentialmodescombiningeyetrackingwithsteadystatevisualevokedpotentialinvirtualrealityenvironment |