Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe
Frequent hospital visits for hand rehabilitation exercises, such as strengthening and opposition exercises, present significant challenges, especially for patients in remote areas. This paper addresses this problem by developing a Rehabilitation Internet-of-Things (RIOT) system that utilizes MediaP...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IIUM Press, International Islamic University Malaysia
2025-01-01
|
| Series: | International Islamic University Malaysia Engineering Journal |
| Subjects: | |
| Online Access: | https://journals.iium.edu.my/ejournal/index.php/iiumej/article/view/3455 |
| _version_ | 1826883563090345984 |
|---|---|
| author | Nurul Hanis Mohd Dhuzuki Ahmad Anwar Zainuddin Nur Anis Sofea Kamarul Zaman Alin Nur Maisarah Ahmad Razmi Wonderful Shammah Kaitane Asmarani Ahmad Puzi Mohd Naqiuddin Johar Maslina Yazid Nor Azlin Mohd Nordin Shahrul Naim Sidek Hasan Firdaus Mohd Zaki |
| author_facet | Nurul Hanis Mohd Dhuzuki Ahmad Anwar Zainuddin Nur Anis Sofea Kamarul Zaman Alin Nur Maisarah Ahmad Razmi Wonderful Shammah Kaitane Asmarani Ahmad Puzi Mohd Naqiuddin Johar Maslina Yazid Nor Azlin Mohd Nordin Shahrul Naim Sidek Hasan Firdaus Mohd Zaki |
| author_sort | Nurul Hanis Mohd Dhuzuki |
| collection | DOAJ |
| description |
Frequent hospital visits for hand rehabilitation exercises, such as strengthening and opposition exercises, present significant challenges, especially for patients in remote areas. This paper addresses this problem by developing a Rehabilitation Internet-of-Things (RIOT) system that utilizes MediaPipe with its pre-trained Deep Learning (DL) to deliver real-time feedback during hand rehabilitation exercises alongside Web Assembly (WASM) for efficient processing. The system's objective is to provide precise, real-time tracking of hand movements, enabling patients to perform exercises at home by maintaining an optimal distance between the camera and hand placement, ensuring ideal room lighting conditions across IoT devices such as mobile phones' front cameras and webcams, while healthcare professionals remotely monitor their progress. The methodology involves the integration of MediaPipe for detecting hand landmarks and adaptive sensitivity algorithms to ensure reliable recognition across different environments, such as varying lighting and hand positions. Future work could incorporate additional deep-learning models like CNNs and RNNs to enhance gesture classification accuracy. Several limitations, including latency and distance sensitivity, are addressed in this system with edge computing alongside adaptive algorithms. The key contributions of this research are as follows: First, developing a real-time and cost-effective solution for remote stroke rehabilitation. Second, accuracy is improved by integrating MediaPipe with deep learning techniques. Lastly, latency issues and accuracy challenges at extended distances are alleviated by employing innovative calibration methods and adaptive adjustments. Initial trials demonstrate promising results, though further testing is required under real-world conditions to validate the system's effectiveness fully.
ABSTRAK: Perjalanan yang kerap ke hospital untuk latihan pemulihan tangan, seperti latihan rawatan fisioterapi telah memberikan cabaran yang besar bagi pesakit yang tinggal di pedalaman. Sistem Pemulihan Internet Benda (RIOT) menggunakan MediaPipe bersama Deep Learning (DL) yang telah dilatih untuk memberikan maklum balas masa nyata semasa latihan pemulihan tangan, serta Web Assembly (WASM) untuk pemprosesan yang cekap, sebagai penyelesaian. Tujuan sistem ini adalah untuk menyediakan penjejakan pergerakan tangan yang tepat dalam masa nyata, yang mampu dijalankan latihan di rumah dengan pemantauan pegawai perubatan untuk meneliti kemajuan mereka dari jarak jauh. Metodologi melibatkan penyatuan MediaPipe untuk mengesan titik penting pada tangan dan algoritma kepekaan suaian untuk memastikan pengiktirafan yang boleh dipercayai dalam pelbagai persekitaran, seperti pencahayaan dan kedudukan tangan. Lonjakan bagi kajian in adalah dapat menggabungkan model DL seperti CNNs dan RNNs untuk meningkatkan ketepatan dan penyusunan isyarat. Sistem ini juga dapat mengurangkan masalah masa pendam dan perubahab jara dengan melaksanakan edge computing dan penyesuaian algoritma. Sumbangan utama kajian ini termasuklah sistem masa nyata yang kos efektif untuk pemulihan strok jarak jauh, peningkatan ketepatan melalui gabungan MediaPipe dan model DL, dan pengurangan masalah masa pendam dan ketepatan jarak yang lebih jauh melalui tentuukur dan suaian algoritma. Percubaan awal telah menunjukkan hasil yang bagus. Walau bagaimanapun, ujian lanjut masih perlu dibuat dalam dunia sebenar untuk menjamin keberkesanan sistem secara keseluruhan.
|
| first_indexed | 2025-02-17T03:47:20Z |
| format | Article |
| id | doaj.art-7dc802b1939542e4938a504cd9de925f |
| institution | Directory Open Access Journal |
| issn | 1511-788X 2289-7860 |
| language | English |
| last_indexed | 2025-02-17T03:47:20Z |
| publishDate | 2025-01-01 |
| publisher | IIUM Press, International Islamic University Malaysia |
| record_format | Article |
| series | International Islamic University Malaysia Engineering Journal |
| spelling | doaj.art-7dc802b1939542e4938a504cd9de925f2025-01-10T12:40:36ZengIIUM Press, International Islamic University MalaysiaInternational Islamic University Malaysia Engineering Journal1511-788X2289-78602025-01-0126110.31436/iiumej.v26i1.3455Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipeNurul Hanis Mohd Dhuzuki0https://orcid.org/0009-0000-7177-3535Ahmad Anwar Zainuddin 1Nur Anis Sofea Kamarul Zaman2https://orcid.org/0009-0004-5542-7471Alin Nur Maisarah Ahmad Razmi3https://orcid.org/0009-0005-3485-2724Wonderful Shammah Kaitane4Asmarani Ahmad Puzi5https://orcid.org/0000-0001-8796-7289Mohd Naqiuddin Johar6Maslina Yazid7Nor Azlin Mohd Nordin8https://orcid.org/0000-0002-0038-1670Shahrul Naim Sidek9https://orcid.org/0000-0002-3204-1347Hasan Firdaus Mohd Zaki10https://orcid.org/0000-0002-7209-4355International Islamic University Malaysia International Islamic University Malaysia International Islamic University Malaysia International Islamic University Malaysia MILA University International Islamic University MalaysiaHospital PutrajayaHospital Shah AlamNational University of Malaysia International Islamic University Malaysia International Islamic University Malaysia Frequent hospital visits for hand rehabilitation exercises, such as strengthening and opposition exercises, present significant challenges, especially for patients in remote areas. This paper addresses this problem by developing a Rehabilitation Internet-of-Things (RIOT) system that utilizes MediaPipe with its pre-trained Deep Learning (DL) to deliver real-time feedback during hand rehabilitation exercises alongside Web Assembly (WASM) for efficient processing. The system's objective is to provide precise, real-time tracking of hand movements, enabling patients to perform exercises at home by maintaining an optimal distance between the camera and hand placement, ensuring ideal room lighting conditions across IoT devices such as mobile phones' front cameras and webcams, while healthcare professionals remotely monitor their progress. The methodology involves the integration of MediaPipe for detecting hand landmarks and adaptive sensitivity algorithms to ensure reliable recognition across different environments, such as varying lighting and hand positions. Future work could incorporate additional deep-learning models like CNNs and RNNs to enhance gesture classification accuracy. Several limitations, including latency and distance sensitivity, are addressed in this system with edge computing alongside adaptive algorithms. The key contributions of this research are as follows: First, developing a real-time and cost-effective solution for remote stroke rehabilitation. Second, accuracy is improved by integrating MediaPipe with deep learning techniques. Lastly, latency issues and accuracy challenges at extended distances are alleviated by employing innovative calibration methods and adaptive adjustments. Initial trials demonstrate promising results, though further testing is required under real-world conditions to validate the system's effectiveness fully. ABSTRAK: Perjalanan yang kerap ke hospital untuk latihan pemulihan tangan, seperti latihan rawatan fisioterapi telah memberikan cabaran yang besar bagi pesakit yang tinggal di pedalaman. Sistem Pemulihan Internet Benda (RIOT) menggunakan MediaPipe bersama Deep Learning (DL) yang telah dilatih untuk memberikan maklum balas masa nyata semasa latihan pemulihan tangan, serta Web Assembly (WASM) untuk pemprosesan yang cekap, sebagai penyelesaian. Tujuan sistem ini adalah untuk menyediakan penjejakan pergerakan tangan yang tepat dalam masa nyata, yang mampu dijalankan latihan di rumah dengan pemantauan pegawai perubatan untuk meneliti kemajuan mereka dari jarak jauh. Metodologi melibatkan penyatuan MediaPipe untuk mengesan titik penting pada tangan dan algoritma kepekaan suaian untuk memastikan pengiktirafan yang boleh dipercayai dalam pelbagai persekitaran, seperti pencahayaan dan kedudukan tangan. Lonjakan bagi kajian in adalah dapat menggabungkan model DL seperti CNNs dan RNNs untuk meningkatkan ketepatan dan penyusunan isyarat. Sistem ini juga dapat mengurangkan masalah masa pendam dan perubahab jara dengan melaksanakan edge computing dan penyesuaian algoritma. Sumbangan utama kajian ini termasuklah sistem masa nyata yang kos efektif untuk pemulihan strok jarak jauh, peningkatan ketepatan melalui gabungan MediaPipe dan model DL, dan pengurangan masalah masa pendam dan ketepatan jarak yang lebih jauh melalui tentuukur dan suaian algoritma. Percubaan awal telah menunjukkan hasil yang bagus. Walau bagaimanapun, ujian lanjut masih perlu dibuat dalam dunia sebenar untuk menjamin keberkesanan sistem secara keseluruhan. https://journals.iium.edu.my/ejournal/index.php/iiumej/article/view/3455Rehabilitation Internet-of-Things (RIOT)MediaPipeDeep Learning (DL)hand gesture recognitionArtificial Intelligence (AI) |
| spellingShingle | Nurul Hanis Mohd Dhuzuki Ahmad Anwar Zainuddin Nur Anis Sofea Kamarul Zaman Alin Nur Maisarah Ahmad Razmi Wonderful Shammah Kaitane Asmarani Ahmad Puzi Mohd Naqiuddin Johar Maslina Yazid Nor Azlin Mohd Nordin Shahrul Naim Sidek Hasan Firdaus Mohd Zaki Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe International Islamic University Malaysia Engineering Journal Rehabilitation Internet-of-Things (RIOT) MediaPipe Deep Learning (DL) hand gesture recognition Artificial Intelligence (AI) |
| title | Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe |
| title_full | Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe |
| title_fullStr | Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe |
| title_full_unstemmed | Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe |
| title_short | Design and Implementation of a Deep Learning-Based Hand Gesture Recognition System for Rehabilitation Internet-of-Things (RIoT) Environments Using MediaPipe |
| title_sort | design and implementation of a deep learning based hand gesture recognition system for rehabilitation internet of things riot environments using mediapipe |
| topic | Rehabilitation Internet-of-Things (RIOT) MediaPipe Deep Learning (DL) hand gesture recognition Artificial Intelligence (AI) |
| url | https://journals.iium.edu.my/ejournal/index.php/iiumej/article/view/3455 |
| work_keys_str_mv | AT nurulhanismohddhuzuki designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT ahmadanwarzainuddin designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT nuranissofeakamarulzaman designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT alinnurmaisarahahmadrazmi designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT wonderfulshammahkaitane designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT asmaraniahmadpuzi designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT mohdnaqiuddinjohar designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT maslinayazid designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT norazlinmohdnordin designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT shahrulnaimsidek designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe AT hasanfirdausmohdzaki designandimplementationofadeeplearningbasedhandgesturerecognitionsystemforrehabilitationinternetofthingsriotenvironmentsusingmediapipe |