RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets
Accurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/11/18/3969 |
| _version_ | 1827725411112452096 |
|---|---|
| author | Wenjun Ruan Wenbo Zhu Zhijia Zhao Kai Wang Qinghua Lu Lufeng Luo Wei-Chang Yeh |
| author_facet | Wenjun Ruan Wenbo Zhu Zhijia Zhao Kai Wang Qinghua Lu Lufeng Luo Wei-Chang Yeh |
| author_sort | Wenjun Ruan |
| collection | DOAJ |
| description | Accurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state detection network of the target. To address this issue, we present a novel visual–tactile fusion model (i.e., RFCT) and provide an incremental dimensional tensor product method for detecting grasping states of weak-stiffness targets. We investigate whether convolutional block attention mechanisms (CBAM) can enhance feature representations by selectively attending to salient visual and tactile cues while suppressing less important information and eliminating redundant information for the initial fusion. We conducted 2250 grasping experiments using 15 weak-stiffness targets. We used 12 targets for training and three for testing. When evaluated on untrained targets, our RFCT model achieved a precision of 82.89%, a recall rate of 82.07%, and an F1 score of 81.65%. We compared RFCT model performance with various combinations of Resnet50 + LSTM and C3D models commonly used in grasping state detection. The experimental results show that our RFCT model significantly outperforms these models. Our proposed method provides accurate grasping state detection and has the potential to provide robust support for robot grasping operations in real-world applications. |
| first_indexed | 2024-03-10T22:29:55Z |
| format | Article |
| id | doaj.art-596c2d1ec71a4848a0a74c8caf47ee6d |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T22:29:55Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-596c2d1ec71a4848a0a74c8caf47ee6d2023-11-19T11:50:07ZengMDPI AGMathematics2227-73902023-09-011118396910.3390/math11183969RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness TargetsWenjun Ruan0Wenbo Zhu1Zhijia Zhao2Kai Wang3Qinghua Lu4Lufeng Luo5Wei-Chang Yeh6School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, ChinaSchool of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, ChinaSchool of Mechanical and Electrical Engineering, Guangzhou University, Guangzhou 510006, ChinaSchool of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, ChinaSchool of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, ChinaSchool of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, ChinaDepartment of Industrial Engineering and Engineering Management, National Tsing Hua University, Hsinchu 30013, TaiwanAccurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state detection network of the target. To address this issue, we present a novel visual–tactile fusion model (i.e., RFCT) and provide an incremental dimensional tensor product method for detecting grasping states of weak-stiffness targets. We investigate whether convolutional block attention mechanisms (CBAM) can enhance feature representations by selectively attending to salient visual and tactile cues while suppressing less important information and eliminating redundant information for the initial fusion. We conducted 2250 grasping experiments using 15 weak-stiffness targets. We used 12 targets for training and three for testing. When evaluated on untrained targets, our RFCT model achieved a precision of 82.89%, a recall rate of 82.07%, and an F1 score of 81.65%. We compared RFCT model performance with various combinations of Resnet50 + LSTM and C3D models commonly used in grasping state detection. The experimental results show that our RFCT model significantly outperforms these models. Our proposed method provides accurate grasping state detection and has the potential to provide robust support for robot grasping operations in real-world applications.https://www.mdpi.com/2227-7390/11/18/3969visual–tactile fusion perceptiontarget grasping state detectiongraspingmultimodal perception |
| spellingShingle | Wenjun Ruan Wenbo Zhu Zhijia Zhao Kai Wang Qinghua Lu Lufeng Luo Wei-Chang Yeh RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets Mathematics visual–tactile fusion perception target grasping state detection grasping multimodal perception |
| title | RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets |
| title_full | RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets |
| title_fullStr | RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets |
| title_full_unstemmed | RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets |
| title_short | RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets |
| title_sort | rfct multimodal sensing enhances grasping state detection for weak stiffness targets |
| topic | visual–tactile fusion perception target grasping state detection grasping multimodal perception |
| url | https://www.mdpi.com/2227-7390/11/18/3969 |
| work_keys_str_mv | AT wenjunruan rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT wenbozhu rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT zhijiazhao rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT kaiwang rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT qinghualu rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT lufengluo rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets AT weichangyeh rfctmultimodalsensingenhancesgraspingstatedetectionforweakstiffnesstargets |