Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system
Computer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Universitas Ahmad Dahlan
2023-07-01
|
| Series: | IJAIN (International Journal of Advances in Intelligent Informatics) |
| Subjects: | |
| Online Access: | https://ijain.org/index.php/IJAIN/article/view/901 |
| _version_ | 1797836012727042048 |
|---|---|
| author | Adnan Rachmat Anom Besari Azhar Aulia Saputra Wei Hong Chin Kurnianingsih Kurnianingsih Naoyuki Kubota |
| author_facet | Adnan Rachmat Anom Besari Azhar Aulia Saputra Wei Hong Chin Kurnianingsih Kurnianingsih Naoyuki Kubota |
| author_sort | Adnan Rachmat Anom Besari |
| collection | DOAJ |
| description | Computer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes a multiscopic CPSS framework by developing hand–object interaction (HOI) based on visual attention. First, we use egocentric vision to extract features from hand posture at the microscopic level. With 94.87% testing accuracy, we use three layers of graph neural network (GNN) based on hand skeletal features to categorize 16 grasp postures. Second, we use a mesoscopic active perception ability to validate the HOI with eye tracking in the task-specific reach-to-grasp cycle. With 90.75% testing accuracy, the distance between the fingertips and the center of an object is used as input to a multi-layer gated recurrent unit based on recurrent neural network architecture. Third, we incorporate visual attention into the cognitive ability for classifying multiple objects at the macroscopic level. In two scenarios with four activities, we use GNN with three convolutional layers to categorize some objects. The outcome demonstrates that the system can successfully separate objects based on related activities. Further research and development are expected to support the CPSS application in independent rehabilitation. |
| first_indexed | 2024-04-09T15:01:45Z |
| format | Article |
| id | doaj.art-5e013bf669654f8cb12da498d340877a |
| institution | Directory Open Access Journal |
| issn | 2442-6571 2548-3161 |
| language | English |
| last_indexed | 2024-04-09T15:01:45Z |
| publishDate | 2023-07-01 |
| publisher | Universitas Ahmad Dahlan |
| record_format | Article |
| series | IJAIN (International Journal of Advances in Intelligent Informatics) |
| spelling | doaj.art-5e013bf669654f8cb12da498d340877a2023-05-01T13:46:21ZengUniversitas Ahmad DahlanIJAIN (International Journal of Advances in Intelligent Informatics)2442-65712548-31612023-07-019218720510.26555/ijain.v9i2.901239Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social systemAdnan Rachmat Anom Besari0Azhar Aulia Saputra1Wei Hong Chin2Kurnianingsih Kurnianingsih3Naoyuki Kubota4Politeknik Elektronika Negeri Surabaya (PENS), Indonesia, and Graduate School of Systems Design,Tokyo Metropolitan UniversityGraduate School of Systems Design, Tokyo Metropolitan UniversityGraduate School of Systems Design, Tokyo Metropolitan UniversityDepartment of Electrical Engineering, Politeknik Negeri SemarangGraduate School of Systems Design, Tokyo Metropolitan UniversityComputer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes a multiscopic CPSS framework by developing hand–object interaction (HOI) based on visual attention. First, we use egocentric vision to extract features from hand posture at the microscopic level. With 94.87% testing accuracy, we use three layers of graph neural network (GNN) based on hand skeletal features to categorize 16 grasp postures. Second, we use a mesoscopic active perception ability to validate the HOI with eye tracking in the task-specific reach-to-grasp cycle. With 90.75% testing accuracy, the distance between the fingertips and the center of an object is used as input to a multi-layer gated recurrent unit based on recurrent neural network architecture. Third, we incorporate visual attention into the cognitive ability for classifying multiple objects at the macroscopic level. In two scenarios with four activities, we use GNN with three convolutional layers to categorize some objects. The outcome demonstrates that the system can successfully separate objects based on related activities. Further research and development are expected to support the CPSS application in independent rehabilitation.https://ijain.org/index.php/IJAIN/article/view/901telemedicinefirst-person visionhand-eye coordinationindependent rehabilitationoccupational therapy |
| spellingShingle | Adnan Rachmat Anom Besari Azhar Aulia Saputra Wei Hong Chin Kurnianingsih Kurnianingsih Naoyuki Kubota Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system IJAIN (International Journal of Advances in Intelligent Informatics) telemedicine first-person vision hand-eye coordination independent rehabilitation occupational therapy |
| title | Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system |
| title_full | Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system |
| title_fullStr | Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system |
| title_full_unstemmed | Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system |
| title_short | Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system |
| title_sort | hand object interaction recognition based on visual attention using multiscopic cyber physical social system |
| topic | telemedicine first-person vision hand-eye coordination independent rehabilitation occupational therapy |
| url | https://ijain.org/index.php/IJAIN/article/view/901 |
| work_keys_str_mv | AT adnanrachmatanombesari handobjectinteractionrecognitionbasedonvisualattentionusingmultiscopiccyberphysicalsocialsystem AT azharauliasaputra handobjectinteractionrecognitionbasedonvisualattentionusingmultiscopiccyberphysicalsocialsystem AT weihongchin handobjectinteractionrecognitionbasedonvisualattentionusingmultiscopiccyberphysicalsocialsystem AT kurnianingsihkurnianingsih handobjectinteractionrecognitionbasedonvisualattentionusingmultiscopiccyberphysicalsocialsystem AT naoyukikubota handobjectinteractionrecognitionbasedonvisualattentionusingmultiscopiccyberphysicalsocialsystem |