Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control
Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatig...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2024-02-01
|
| Series: | Frontiers in Robotics and AI |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frobt.2024.1312554/full |
| _version_ | 1797294171044708352 |
|---|---|
| author | Mehrshad Zandigohar Mo Han Mohammadreza Sharif Sezen Yağmur Günay Mariusz P. Furmanek Mariusz P. Furmanek Mathew Yarossi Paolo Bonato Cagdas Onal Taşkın Padır Deniz Erdoğmuş Gunar Schirner |
| author_facet | Mehrshad Zandigohar Mo Han Mohammadreza Sharif Sezen Yağmur Günay Mariusz P. Furmanek Mariusz P. Furmanek Mathew Yarossi Paolo Bonato Cagdas Onal Taşkın Padır Deniz Erdoğmuş Gunar Schirner |
| author_sort | Mehrshad Zandigohar |
| collection | DOAJ |
| description | Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities.Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components.Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%.Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time. |
| first_indexed | 2024-03-07T21:26:22Z |
| format | Article |
| id | doaj.art-f1c3baaf51ee46a7a73971adf623eff3 |
| institution | Directory Open Access Journal |
| issn | 2296-9144 |
| language | English |
| last_indexed | 2024-03-07T21:26:22Z |
| publishDate | 2024-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Robotics and AI |
| spelling | doaj.art-f1c3baaf51ee46a7a73971adf623eff32024-02-27T04:39:12ZengFrontiers Media S.A.Frontiers in Robotics and AI2296-91442024-02-011110.3389/frobt.2024.13125541312554Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand controlMehrshad Zandigohar0Mo Han1Mohammadreza Sharif2Sezen Yağmur Günay3Mariusz P. Furmanek4Mariusz P. Furmanek5Mathew Yarossi6Paolo Bonato7Cagdas Onal8Taşkın Padır9Deniz Erdoğmuş10Gunar Schirner11Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Physical Therapy, Movement and Rehabilitation Sciences, Northeastern University, Boston, MA, United StatesInstitute of Sport Sciences, Academy of Physical Education in Katowice, Katowice, PolandDepartment of Physical Therapy, Movement and Rehabilitation Sciences, Northeastern University, Boston, MA, United StatesMotion Analysis Lab, Spaulding Rehabilitation Hospital, Charlestown, MA, United StatesSoft Robotics Lab, Worcester Polytechnic Institute, Worcester, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesDepartment of Electrical and Computer Engineering, Northeastern University, Boston, MA, United StatesObjective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities.Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components.Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%.Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.https://www.frontiersin.org/articles/10.3389/frobt.2024.1312554/fulldatasetEMGgrasp detectionneural networksrobotic prosthetic hand |
| spellingShingle | Mehrshad Zandigohar Mo Han Mohammadreza Sharif Sezen Yağmur Günay Mariusz P. Furmanek Mariusz P. Furmanek Mathew Yarossi Paolo Bonato Cagdas Onal Taşkın Padır Deniz Erdoğmuş Gunar Schirner Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control Frontiers in Robotics and AI dataset EMG grasp detection neural networks robotic prosthetic hand |
| title | Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control |
| title_full | Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control |
| title_fullStr | Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control |
| title_full_unstemmed | Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control |
| title_short | Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control |
| title_sort | multimodal fusion of emg and vision for human grasp intent inference in prosthetic hand control |
| topic | dataset EMG grasp detection neural networks robotic prosthetic hand |
| url | https://www.frontiersin.org/articles/10.3389/frobt.2024.1312554/full |
| work_keys_str_mv | AT mehrshadzandigohar multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT mohan multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT mohammadrezasharif multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT sezenyagmurgunay multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT mariuszpfurmanek multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT mariuszpfurmanek multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT mathewyarossi multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT paolobonato multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT cagdasonal multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT taskınpadır multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT denizerdogmus multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol AT gunarschirner multimodalfusionofemgandvisionforhumangraspintentinferenceinprosthetichandcontrol |