Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention
This paper introduces a multiscopic cyber-physical-social system (CPSS) to bridge the gap between independent rehabilitation in physical and cognitive aspects. Specifically, we focus on hand–object interaction (HOI) recognition with visual attention for the block-design test (BDT). The pr...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10143635/ |
| _version_ | 1797803148001148928 |
|---|---|
| author | Adnan Rachmat Anom Besari Azhar Aulia Saputra Takenori Obo Kurnianingsih Naoyuki Kubota |
| author_facet | Adnan Rachmat Anom Besari Azhar Aulia Saputra Takenori Obo Kurnianingsih Naoyuki Kubota |
| author_sort | Adnan Rachmat Anom Besari |
| collection | DOAJ |
| description | This paper introduces a multiscopic cyber-physical-social system (CPSS) to bridge the gap between independent rehabilitation in physical and cognitive aspects. Specifically, we focus on hand–object interaction (HOI) recognition with visual attention for the block-design test (BDT). The proposed framework utilizes three levels which consist of microscopic, mesoscopic, and macroscopic models. In the microscopic model, a hand-tracking vision captures hand-skeletal data and finger joint angle features, enabling the estimation of physical hand postures. In the mesoscopic model, an egocentric vision with an eye tracker records to hand and eye movements, allowing for the symbolic representation of hand-eye coordination through hand gestures and visual attention focus during the test. An evaluation vision system employs color feature classification in the macroscopic model to determine whether the design matches the given task. Through the first eight designs of WAIS-IV BDT with two scenarios, the system successfully measures human behavior from the physical to the cognitive domain. The experiment involving eight healthy participants investigates the relationship between physical measurement and cognitive evaluation. Regression and correlation analyses between the dominant and non-dominant hands reveal that evaluation indices (task completion time, skewness-kurtosis of hand posture, attention to pattern and blocks) can indicate improvement during BDT. The outcomes of this study have significant implications for clinicians and researchers, providing valuable information that is typically unavailable in clinical settings. The proposed multiscopic CPSS framework holds promise for advancing independent rehabilitation practices. Code and datasets are available online at <uri>https://github.com/anom-tmu/bdt-multiscopic</uri>. |
| first_indexed | 2024-03-13T05:16:29Z |
| format | Article |
| id | doaj.art-c14c6aac8079490fac58804fe791c95c |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-13T05:16:29Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-c14c6aac8079490fac58804fe791c95c2023-06-15T23:00:23ZengIEEEIEEE Access2169-35362023-01-0111581885820810.1109/ACCESS.2023.328287610143635Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual AttentionAdnan Rachmat Anom Besari0https://orcid.org/0000-0001-5916-5019Azhar Aulia Saputra1https://orcid.org/0000-0001-9027-8935Takenori Obo2https://orcid.org/0009-0009-0318-470X Kurnianingsih3https://orcid.org/0000-0001-7339-7449Naoyuki Kubota4https://orcid.org/0000-0001-8829-037XGraduate School of Systems Design, Tokyo Metropolitan University, Tokyo, JapanGraduate School of Systems Design, Tokyo Metropolitan University, Tokyo, JapanGraduate School of Systems Design, Tokyo Metropolitan University, Tokyo, JapanDepartment of Electrical Engineering, Politeknik Negeri Semarang, Semarang, IndonesiaGraduate School of Systems Design, Tokyo Metropolitan University, Tokyo, JapanThis paper introduces a multiscopic cyber-physical-social system (CPSS) to bridge the gap between independent rehabilitation in physical and cognitive aspects. Specifically, we focus on hand–object interaction (HOI) recognition with visual attention for the block-design test (BDT). The proposed framework utilizes three levels which consist of microscopic, mesoscopic, and macroscopic models. In the microscopic model, a hand-tracking vision captures hand-skeletal data and finger joint angle features, enabling the estimation of physical hand postures. In the mesoscopic model, an egocentric vision with an eye tracker records to hand and eye movements, allowing for the symbolic representation of hand-eye coordination through hand gestures and visual attention focus during the test. An evaluation vision system employs color feature classification in the macroscopic model to determine whether the design matches the given task. Through the first eight designs of WAIS-IV BDT with two scenarios, the system successfully measures human behavior from the physical to the cognitive domain. The experiment involving eight healthy participants investigates the relationship between physical measurement and cognitive evaluation. Regression and correlation analyses between the dominant and non-dominant hands reveal that evaluation indices (task completion time, skewness-kurtosis of hand posture, attention to pattern and blocks) can indicate improvement during BDT. The outcomes of this study have significant implications for clinicians and researchers, providing valuable information that is typically unavailable in clinical settings. The proposed multiscopic CPSS framework holds promise for advancing independent rehabilitation practices. Code and datasets are available online at <uri>https://github.com/anom-tmu/bdt-multiscopic</uri>.https://ieeexplore.ieee.org/document/10143635/Kohs blocks designself-rehabilitationvisual attentionhand-eye coordination |
| spellingShingle | Adnan Rachmat Anom Besari Azhar Aulia Saputra Takenori Obo Kurnianingsih Naoyuki Kubota Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention IEEE Access Kohs blocks design self-rehabilitation visual attention hand-eye coordination |
| title | Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention |
| title_full | Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention |
| title_fullStr | Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention |
| title_full_unstemmed | Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention |
| title_short | Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention |
| title_sort | multiscopic cpss for independent block design test based on hand x2013 object interaction recognition with visual attention |
| topic | Kohs blocks design self-rehabilitation visual attention hand-eye coordination |
| url | https://ieeexplore.ieee.org/document/10143635/ |
| work_keys_str_mv | AT adnanrachmatanombesari multiscopiccpssforindependentblockdesigntestbasedonhandx2013objectinteractionrecognitionwithvisualattention AT azharauliasaputra multiscopiccpssforindependentblockdesigntestbasedonhandx2013objectinteractionrecognitionwithvisualattention AT takenoriobo multiscopiccpssforindependentblockdesigntestbasedonhandx2013objectinteractionrecognitionwithvisualattention AT kurnianingsih multiscopiccpssforindependentblockdesigntestbasedonhandx2013objectinteractionrecognitionwithvisualattention AT naoyukikubota multiscopiccpssforindependentblockdesigntestbasedonhandx2013objectinteractionrecognitionwithvisualattention |