A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers
Accurate hand motion intention recognition is essential for the intuitive control of intelligent prosthetic hands and other human-machine interaction systems. Sonomyography, which can detect the changes in muscle morphology and structure precisely, is a promising signal source for fine hand movement...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10318819/ |
| _version_ | 1827650408377483264 |
|---|---|
| author | Xinhao Peng Yan Liu Fangning Tan Weicen Chen Zhiyuan Liu Teng Ma Xiangxin Li Guanglin Li |
| author_facet | Xinhao Peng Yan Liu Fangning Tan Weicen Chen Zhiyuan Liu Teng Ma Xiangxin Li Guanglin Li |
| author_sort | Xinhao Peng |
| collection | DOAJ |
| description | Accurate hand motion intention recognition is essential for the intuitive control of intelligent prosthetic hands and other human-machine interaction systems. Sonomyography, which can detect the changes in muscle morphology and structure precisely, is a promising signal source for fine hand movement recognition. However, sonomyography measured by traditional rigid ultrasound probes may suffer from poor acoustic coupling because the rigid probe surfaces cannot accommodate the curvilinear shape of the human body, particularly in the case of small and irregular residual limbs in amputees. In this study, we used a self-designed lightweight, flexible, and wearable ultrasound transducer to acquire muscle ultrasound images, and proposed a sonomyography transformer (SMGT) model for simultaneous recognition of hand movements and force levels. The performance of SMGT was systematically compared to two commonly used image processing methods, HOG and Gray Gradient, as well as a deep CNN model, in simultaneously recognizing ten classes of hand/finger movements and three force levels. Additionally, ten subjects including seven non-disabled subjects and three trans-radial amputees who are the end users of prosthetic hands were recruited to evaluate the effectiveness of SMGT. Results showed that our proposed method achieved average classification accuracies of 98.4% ± 0.6% and 96.2% ± 3.0% in non-disabled subjects and amputee subjects, respectively, which are much higher than those of other methods. This study provided a valuable approach for ultrasound-based hand motion recognition that may promote the applications of intelligent prosthetic hands. |
| first_indexed | 2024-03-09T20:16:46Z |
| format | Article |
| id | doaj.art-b4550ee46e77496d827c4d444928f76b |
| institution | Directory Open Access Journal |
| issn | 1558-0210 |
| language | English |
| last_indexed | 2024-03-09T20:16:46Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| spelling | doaj.art-b4550ee46e77496d827c4d444928f76b2023-11-24T00:00:15ZengIEEEIEEE Transactions on Neural Systems and Rehabilitation Engineering1558-02102023-01-01314580459010.1109/TNSRE.2023.333300810318819A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound TransducersXinhao Peng0Yan Liu1Fangning Tan2https://orcid.org/0000-0002-4391-1854Weicen Chen3Zhiyuan Liu4https://orcid.org/0000-0001-9231-8195Teng Ma5https://orcid.org/0000-0002-5849-2138Xiangxin Li6https://orcid.org/0000-0003-1232-6182Guanglin Li7https://orcid.org/0000-0001-9016-2617CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaCAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaCAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaInstitute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaCAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaInstitute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaCAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaCAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaAccurate hand motion intention recognition is essential for the intuitive control of intelligent prosthetic hands and other human-machine interaction systems. Sonomyography, which can detect the changes in muscle morphology and structure precisely, is a promising signal source for fine hand movement recognition. However, sonomyography measured by traditional rigid ultrasound probes may suffer from poor acoustic coupling because the rigid probe surfaces cannot accommodate the curvilinear shape of the human body, particularly in the case of small and irregular residual limbs in amputees. In this study, we used a self-designed lightweight, flexible, and wearable ultrasound transducer to acquire muscle ultrasound images, and proposed a sonomyography transformer (SMGT) model for simultaneous recognition of hand movements and force levels. The performance of SMGT was systematically compared to two commonly used image processing methods, HOG and Gray Gradient, as well as a deep CNN model, in simultaneously recognizing ten classes of hand/finger movements and three force levels. Additionally, ten subjects including seven non-disabled subjects and three trans-radial amputees who are the end users of prosthetic hands were recruited to evaluate the effectiveness of SMGT. Results showed that our proposed method achieved average classification accuracies of 98.4% ± 0.6% and 96.2% ± 3.0% in non-disabled subjects and amputee subjects, respectively, which are much higher than those of other methods. This study provided a valuable approach for ultrasound-based hand motion recognition that may promote the applications of intelligent prosthetic hands.https://ieeexplore.ieee.org/document/10318819/Sonomyographywearable muscle ultrasoundhand motion intention recognitiontransformertrans-radial amputees |
| spellingShingle | Xinhao Peng Yan Liu Fangning Tan Weicen Chen Zhiyuan Liu Teng Ma Xiangxin Li Guanglin Li A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers IEEE Transactions on Neural Systems and Rehabilitation Engineering Sonomyography wearable muscle ultrasound hand motion intention recognition transformer trans-radial amputees |
| title | A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers |
| title_full | A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers |
| title_fullStr | A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers |
| title_full_unstemmed | A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers |
| title_short | A Novel Transformer-Based Approach for Simultaneous Recognition of Hand Movements and Force Levels in Amputees Using Flexible Ultrasound Transducers |
| title_sort | novel transformer based approach for simultaneous recognition of hand movements and force levels in amputees using flexible ultrasound transducers |
| topic | Sonomyography wearable muscle ultrasound hand motion intention recognition transformer trans-radial amputees |
| url | https://ieeexplore.ieee.org/document/10318819/ |
| work_keys_str_mv | AT xinhaopeng anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT yanliu anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT fangningtan anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT weicenchen anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT zhiyuanliu anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT tengma anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT xiangxinli anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT guanglinli anoveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT xinhaopeng noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT yanliu noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT fangningtan noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT weicenchen noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT zhiyuanliu noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT tengma noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT xiangxinli noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers AT guanglinli noveltransformerbasedapproachforsimultaneousrecognitionofhandmovementsandforcelevelsinamputeesusingflexibleultrasoundtransducers |