A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics
For humans and animals to recognise an object, the integration of multiple sensing methods is essential when one sensing modality is only able to acquire limited information. Among the many sensing modalities, vision has been intensively studied and proven to have superior performance for many probl...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Biomimetics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2313-7673/8/1/86 |
| _version_ | 1827751076358520832 |
|---|---|
| author | Teng Sun Zhe Zhang Zhonghua Miao Wen Zhang |
| author_facet | Teng Sun Zhe Zhang Zhonghua Miao Wen Zhang |
| author_sort | Teng Sun |
| collection | DOAJ |
| description | For humans and animals to recognise an object, the integration of multiple sensing methods is essential when one sensing modality is only able to acquire limited information. Among the many sensing modalities, vision has been intensively studied and proven to have superior performance for many problems. Nevertheless, there are many problems which are difficult to solve by solitary vision, such as in a dark environment or for objects with a similar outlook but different inclusions. Haptic sensing is another commonly used means of perception, which can provide local contact information and physical features that are difficult to obtain by vision. Therefore, the fusion of vision and touch is beneficial to improve the robustness of object perception. To address this, an end-to-end visual–haptic fusion perceptual method has been proposed. In particular, the YOLO deep network is used to extract vision features, while haptic explorations are used to extract haptic features. Then, visual and haptic features are aggregated using a graph convolutional network, and the object is recognised based on a multi-layer perceptron. Experimental results show that the proposed method excels in distinguishing soft objects that have similar appearance but varied interior fillers, comparing a simple convolutional network and a Bayesian filter. The resultant average recognition accuracy was improved to 0.95 from vision only (mAP is 0.502). Moreover, the extracted physical features could be further used for manipulation tasks targeting soft objects. |
| first_indexed | 2024-03-11T06:52:13Z |
| format | Article |
| id | doaj.art-153917766d98454eb9845092316400d0 |
| institution | Directory Open Access Journal |
| issn | 2313-7673 |
| language | English |
| last_indexed | 2024-03-11T06:52:13Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Biomimetics |
| spelling | doaj.art-153917766d98454eb9845092316400d02023-11-17T09:50:06ZengMDPI AGBiomimetics2313-76732023-02-01818610.3390/biomimetics8010086A Recognition Method for Soft Objects Based on the Fusion of Vision and HapticsTeng Sun0Zhe Zhang1Zhonghua Miao2Wen Zhang3Intelligent Equipment and Robotics Lab, Department of Automation, School of Mechatronic Engineering and Automation, Shanghai University, Shangda Street No. 99, Baoshan District, Shanghai 200444, ChinaIntelligent Equipment and Robotics Lab, Department of Automation, School of Mechatronic Engineering and Automation, Shanghai University, Shangda Street No. 99, Baoshan District, Shanghai 200444, ChinaIntelligent Equipment and Robotics Lab, Department of Automation, School of Mechatronic Engineering and Automation, Shanghai University, Shangda Street No. 99, Baoshan District, Shanghai 200444, ChinaIntelligent Equipment and Robotics Lab, Department of Automation, School of Mechatronic Engineering and Automation, Shanghai University, Shangda Street No. 99, Baoshan District, Shanghai 200444, ChinaFor humans and animals to recognise an object, the integration of multiple sensing methods is essential when one sensing modality is only able to acquire limited information. Among the many sensing modalities, vision has been intensively studied and proven to have superior performance for many problems. Nevertheless, there are many problems which are difficult to solve by solitary vision, such as in a dark environment or for objects with a similar outlook but different inclusions. Haptic sensing is another commonly used means of perception, which can provide local contact information and physical features that are difficult to obtain by vision. Therefore, the fusion of vision and touch is beneficial to improve the robustness of object perception. To address this, an end-to-end visual–haptic fusion perceptual method has been proposed. In particular, the YOLO deep network is used to extract vision features, while haptic explorations are used to extract haptic features. Then, visual and haptic features are aggregated using a graph convolutional network, and the object is recognised based on a multi-layer perceptron. Experimental results show that the proposed method excels in distinguishing soft objects that have similar appearance but varied interior fillers, comparing a simple convolutional network and a Bayesian filter. The resultant average recognition accuracy was improved to 0.95 from vision only (mAP is 0.502). Moreover, the extracted physical features could be further used for manipulation tasks targeting soft objects.https://www.mdpi.com/2313-7673/8/1/86deep networkhaptic explorationgraph neural networksensor fusionobject perception |
| spellingShingle | Teng Sun Zhe Zhang Zhonghua Miao Wen Zhang A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics Biomimetics deep network haptic exploration graph neural network sensor fusion object perception |
| title | A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics |
| title_full | A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics |
| title_fullStr | A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics |
| title_full_unstemmed | A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics |
| title_short | A Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics |
| title_sort | recognition method for soft objects based on the fusion of vision and haptics |
| topic | deep network haptic exploration graph neural network sensor fusion object perception |
| url | https://www.mdpi.com/2313-7673/8/1/86 |
| work_keys_str_mv | AT tengsun arecognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT zhezhang arecognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT zhonghuamiao arecognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT wenzhang arecognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT tengsun recognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT zhezhang recognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT zhonghuamiao recognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics AT wenzhang recognitionmethodforsoftobjectsbasedonthefusionofvisionandhaptics |