Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection
In this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/7/2280 |
| _version_ | 1827696869775507456 |
|---|---|
| author | Ching-Chang Wong Li-Yu Yeh Chih-Cheng Liu Chi-Yi Tsai Hisasuki Aoyama |
| author_facet | Ching-Chang Wong Li-Yu Yeh Chih-Cheng Liu Chi-Yi Tsai Hisasuki Aoyama |
| author_sort | Ching-Chang Wong |
| collection | DOAJ |
| description | In this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint detection system; and (2) manipulation planning system for object re-orientation. In the 3D keypoint detection system, an RGB-D camera is used to obtain the information of the environment and can generate 3D keypoints of the target object as inputs to represent its corresponding position and pose. This process simplifies the 3D model representation so that the manipulation planning for object re-orientation can be executed in a category-level manner by adding various training data of the object in the training phase. In addition, 3D suction points in both the object’s current and expected poses are also generated as the inputs of the next operation stage. During the next stage, Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is used for preliminary object detection and object image. The highest confidence index image is selected as the input of the semantic segmentation system in order to classify each pixel in the picture for the corresponding pack unit of the object. In addition, after using a convolutional neural network for semantic segmentation, the Conditional Random Fields (CRFs) method is used to perform several iterations to obtain a more accurate result of object recognition. When the target object is segmented into the pack units of image process, the center position of each pack unit can be obtained. Then, a normal vector of each pack unit’s center points is generated by the depth image information and pose of the object, which can be obtained by connecting the center points of each pack unit. In the manipulation planning system for object re-orientation, the pose of the object and the normal vector of each pack unit are first converted into the working coordinate system of the robot manipulator. Then, according to the current and expected pose of the object, the spherical linear interpolation (Slerp) algorithm is used to generate a series of movements in the workspace for object re-orientation on the robot manipulator. In addition, the pose of the object is adjusted on the z-axis of the object’s geodetic coordinate system based on the image features on the surface of the object, so that the pose of the placed object can approach the desired pose. Finally, a robot manipulator and a vacuum suction cup made by the laboratory are used to verify that the proposed system can indeed complete the planned task of object re-orientation. |
| first_indexed | 2024-03-10T12:56:24Z |
| format | Article |
| id | doaj.art-b22549e906b247ed8cdcf5f145ca8371 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T12:56:24Z |
| publishDate | 2021-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-b22549e906b247ed8cdcf5f145ca83712023-11-21T11:54:19ZengMDPI AGSensors1424-82202021-03-01217228010.3390/s21072280Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint DetectionChing-Chang Wong0Li-Yu Yeh1Chih-Cheng Liu2Chi-Yi Tsai3Hisasuki Aoyama4Department of Electrical and Computer Engineering, Tamkang University, New Taipei City 25137, TaiwanDepartment of Electrical and Computer Engineering, Tamkang University, New Taipei City 25137, TaiwanDepartment of Electrical and Computer Engineering, Tamkang University, New Taipei City 25137, TaiwanDepartment of Electrical and Computer Engineering, Tamkang University, New Taipei City 25137, TaiwanDepartment of Mechanical and Intelligent Systems Engineering, University of Electro-Communications, Tokyo 182-8585, JapanIn this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint detection system; and (2) manipulation planning system for object re-orientation. In the 3D keypoint detection system, an RGB-D camera is used to obtain the information of the environment and can generate 3D keypoints of the target object as inputs to represent its corresponding position and pose. This process simplifies the 3D model representation so that the manipulation planning for object re-orientation can be executed in a category-level manner by adding various training data of the object in the training phase. In addition, 3D suction points in both the object’s current and expected poses are also generated as the inputs of the next operation stage. During the next stage, Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is used for preliminary object detection and object image. The highest confidence index image is selected as the input of the semantic segmentation system in order to classify each pixel in the picture for the corresponding pack unit of the object. In addition, after using a convolutional neural network for semantic segmentation, the Conditional Random Fields (CRFs) method is used to perform several iterations to obtain a more accurate result of object recognition. When the target object is segmented into the pack units of image process, the center position of each pack unit can be obtained. Then, a normal vector of each pack unit’s center points is generated by the depth image information and pose of the object, which can be obtained by connecting the center points of each pack unit. In the manipulation planning system for object re-orientation, the pose of the object and the normal vector of each pack unit are first converted into the working coordinate system of the robot manipulator. Then, according to the current and expected pose of the object, the spherical linear interpolation (Slerp) algorithm is used to generate a series of movements in the workspace for object re-orientation on the robot manipulator. In addition, the pose of the object is adjusted on the z-axis of the object’s geodetic coordinate system based on the image features on the surface of the object, so that the pose of the placed object can approach the desired pose. Finally, a robot manipulator and a vacuum suction cup made by the laboratory are used to verify that the proposed system can indeed complete the planned task of object re-orientation.https://www.mdpi.com/1424-8220/21/7/2280object re-orientationpick-and-placeMask R-CNNsemantic segmentation3D keypoint detection |
| spellingShingle | Ching-Chang Wong Li-Yu Yeh Chih-Cheng Liu Chi-Yi Tsai Hisasuki Aoyama Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection Sensors object re-orientation pick-and-place Mask R-CNN semantic segmentation 3D keypoint detection |
| title | Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection |
| title_full | Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection |
| title_fullStr | Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection |
| title_full_unstemmed | Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection |
| title_short | Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection |
| title_sort | manipulation planning for object re orientation based on semantic segmentation keypoint detection |
| topic | object re-orientation pick-and-place Mask R-CNN semantic segmentation 3D keypoint detection |
| url | https://www.mdpi.com/1424-8220/21/7/2280 |
| work_keys_str_mv | AT chingchangwong manipulationplanningforobjectreorientationbasedonsemanticsegmentationkeypointdetection AT liyuyeh manipulationplanningforobjectreorientationbasedonsemanticsegmentationkeypointdetection AT chihchengliu manipulationplanningforobjectreorientationbasedonsemanticsegmentationkeypointdetection AT chiyitsai manipulationplanningforobjectreorientationbasedonsemanticsegmentationkeypointdetection AT hisasukiaoyama manipulationplanningforobjectreorientationbasedonsemanticsegmentationkeypointdetection |