A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking
An apple-picking robot is now the most widely accepted method in the substitution of low-efficiency and high-cost labor-intensive apple harvesting. Although most current research on apple-picking robots works well in the laboratory, most of them are unworkable in an orchard environment due to unsati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-05-01
|
| Series: | Agronomy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-4395/13/6/1469 |
| _version_ | 1797596588619595776 |
|---|---|
| author | Huijun Zhang Chunhong Tang Xiaoming Sun Longsheng Fu |
| author_facet | Huijun Zhang Chunhong Tang Xiaoming Sun Longsheng Fu |
| author_sort | Huijun Zhang |
| collection | DOAJ |
| description | An apple-picking robot is now the most widely accepted method in the substitution of low-efficiency and high-cost labor-intensive apple harvesting. Although most current research on apple-picking robots works well in the laboratory, most of them are unworkable in an orchard environment due to unsatisfied apple positioning performance. In general, an accurate, fast, and widely used apple positioning method for an apple-picking robot remains lacking. Some positioning methods with detection-based deep learning reached an acceptable performance in some orchards. However, apples occluded by apples, leaves, and branches are ignored in these methods with detection-based deep learning. Therefore, an apple binocular positioning method based on a Mask Region Convolutional Neural Network (Mask R-CNN, an instance segmentation network) was developed to achieve better apple positioning. A binocular camera (Bumblebee XB3) was adapted to capture binocular images of apples. After that, a Mask R-CNN was applied to implement instance segmentation of apple binocular images. Then, template matching with a parallel polar line constraint was applied for the stereo matching of apples. Finally, four feature point pairs of apples from binocular images were selected to calculate disparity and depth. The trained Mask R-CNN reached a detection and segmentation intersection over union (<i>IoU</i>) of 80.11% and 84.39%, respectively. The coefficient of variation (<i>CoV</i>) and positioning accuracy (<i>PA</i>) of binocular positioning were 5.28 mm and 99.49%, respectively. The research developed a new method to fulfill binocular positioning with a segmentation-based neural network. |
| first_indexed | 2024-03-11T02:53:20Z |
| format | Article |
| id | doaj.art-f23319d4e11f4070bdeff446bcf5d4e3 |
| institution | Directory Open Access Journal |
| issn | 2073-4395 |
| language | English |
| last_indexed | 2024-03-11T02:53:20Z |
| publishDate | 2023-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Agronomy |
| spelling | doaj.art-f23319d4e11f4070bdeff446bcf5d4e32023-11-18T08:53:42ZengMDPI AGAgronomy2073-43952023-05-01136146910.3390/agronomy13061469A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic PickingHuijun Zhang0Chunhong Tang1Xiaoming Sun2Longsheng Fu3College of Environmental Resources, Chongqing Technology and Business University, Chongqing 400067, ChinaCollege of Environmental Resources, Chongqing Technology and Business University, Chongqing 400067, ChinaCollege of Mechanical and Electronic Engineering, Northwest A&F University, Yangling 712100, ChinaCollege of Mechanical and Electronic Engineering, Northwest A&F University, Yangling 712100, ChinaAn apple-picking robot is now the most widely accepted method in the substitution of low-efficiency and high-cost labor-intensive apple harvesting. Although most current research on apple-picking robots works well in the laboratory, most of them are unworkable in an orchard environment due to unsatisfied apple positioning performance. In general, an accurate, fast, and widely used apple positioning method for an apple-picking robot remains lacking. Some positioning methods with detection-based deep learning reached an acceptable performance in some orchards. However, apples occluded by apples, leaves, and branches are ignored in these methods with detection-based deep learning. Therefore, an apple binocular positioning method based on a Mask Region Convolutional Neural Network (Mask R-CNN, an instance segmentation network) was developed to achieve better apple positioning. A binocular camera (Bumblebee XB3) was adapted to capture binocular images of apples. After that, a Mask R-CNN was applied to implement instance segmentation of apple binocular images. Then, template matching with a parallel polar line constraint was applied for the stereo matching of apples. Finally, four feature point pairs of apples from binocular images were selected to calculate disparity and depth. The trained Mask R-CNN reached a detection and segmentation intersection over union (<i>IoU</i>) of 80.11% and 84.39%, respectively. The coefficient of variation (<i>CoV</i>) and positioning accuracy (<i>PA</i>) of binocular positioning were 5.28 mm and 99.49%, respectively. The research developed a new method to fulfill binocular positioning with a segmentation-based neural network.https://www.mdpi.com/2073-4395/13/6/1469apple binocular positioninginstance segmentationMask R-CNNtemplate matchingstereo matching |
| spellingShingle | Huijun Zhang Chunhong Tang Xiaoming Sun Longsheng Fu A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking Agronomy apple binocular positioning instance segmentation Mask R-CNN template matching stereo matching |
| title | A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking |
| title_full | A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking |
| title_fullStr | A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking |
| title_full_unstemmed | A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking |
| title_short | A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking |
| title_sort | refined apple binocular positioning method with segmentation based deep learning for robotic picking |
| topic | apple binocular positioning instance segmentation Mask R-CNN template matching stereo matching |
| url | https://www.mdpi.com/2073-4395/13/6/1469 |
| work_keys_str_mv | AT huijunzhang arefinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT chunhongtang arefinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT xiaomingsun arefinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT longshengfu arefinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT huijunzhang refinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT chunhongtang refinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT xiaomingsun refinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking AT longshengfu refinedapplebinocularpositioningmethodwithsegmentationbaseddeeplearningforroboticpicking |