Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach
Since the mature green tomatoes have color similar to branches and leaves, some are shaded by branches and leaves, and overlapped by other tomatoes, the accurate detection and location of these tomatoes is rather difficult. This paper proposes to use the Mask R-CNN algorithm for the detection and se...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/23/7842 |
| _version_ | 1797507224262672384 |
|---|---|
| author | Linlu Zu Yanping Zhao Jiuqin Liu Fei Su Yan Zhang Pingzeng Liu |
| author_facet | Linlu Zu Yanping Zhao Jiuqin Liu Fei Su Yan Zhang Pingzeng Liu |
| author_sort | Linlu Zu |
| collection | DOAJ |
| description | Since the mature green tomatoes have color similar to branches and leaves, some are shaded by branches and leaves, and overlapped by other tomatoes, the accurate detection and location of these tomatoes is rather difficult. This paper proposes to use the Mask R-CNN algorithm for the detection and segmentation of mature green tomatoes. A mobile robot is designed to collect images round-the-clock and with different conditions in the whole greenhouse, thus, to make sure the captured dataset are not only objects with the interest of users. After the training process, RestNet50-FPN is selected as the backbone network. Then, the feature map is trained through the region proposal network to generate the region of interest (ROI), and the ROIAlign bilinear interpolation is used to calculate the target region, such that the corresponding region in the feature map is pooled to a fixed size based on the position coordinates of the preselection box. Finally, the detection and segmentation of mature green tomatoes is realized by the parallel actions of ROI target categories, bounding box regression and mask. When the Intersection over Union is equal to 0.5, the performance of the trained model is the best. The experimental results show that the F1-Score of bounding box and mask region all achieve 92.0%. The image acquisition processes are fully unobservable, without any user preselection, which are a highly heterogenic mix, the selected Mask R-CNN algorithm could also accurately detect mature green tomatoes. The performance of this proposed model in a real greenhouse harvesting environment is also evaluated, thus facilitating the direct application in a tomato harvesting robot. |
| first_indexed | 2024-03-10T04:45:29Z |
| format | Article |
| id | doaj.art-7a78e9789f9c49c6bc70c8cde9991483 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T04:45:29Z |
| publishDate | 2021-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-7a78e9789f9c49c6bc70c8cde99914832023-11-23T02:59:57ZengMDPI AGSensors1424-82202021-11-012123784210.3390/s21237842Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition ApproachLinlu Zu0Yanping Zhao1Jiuqin Liu2Fei Su3Yan Zhang4Pingzeng Liu5College of Mechanical and Electronic Engineering, Shandong Agricultural University, Tai’an 271018, ChinaCollege of Mechanical and Electronic Engineering, Shandong Agricultural University, Tai’an 271018, ChinaCollege of Mechanical and Electronic Engineering, Shandong Agricultural University, Tai’an 271018, ChinaCollege of Mechanical and Electronic Engineering, Shandong Agricultural University, Tai’an 271018, ChinaCollege of Information Science and Engineering, Shandong Agricultural University, Tai’an 271018, ChinaCollege of Information Science and Engineering, Shandong Agricultural University, Tai’an 271018, ChinaSince the mature green tomatoes have color similar to branches and leaves, some are shaded by branches and leaves, and overlapped by other tomatoes, the accurate detection and location of these tomatoes is rather difficult. This paper proposes to use the Mask R-CNN algorithm for the detection and segmentation of mature green tomatoes. A mobile robot is designed to collect images round-the-clock and with different conditions in the whole greenhouse, thus, to make sure the captured dataset are not only objects with the interest of users. After the training process, RestNet50-FPN is selected as the backbone network. Then, the feature map is trained through the region proposal network to generate the region of interest (ROI), and the ROIAlign bilinear interpolation is used to calculate the target region, such that the corresponding region in the feature map is pooled to a fixed size based on the position coordinates of the preselection box. Finally, the detection and segmentation of mature green tomatoes is realized by the parallel actions of ROI target categories, bounding box regression and mask. When the Intersection over Union is equal to 0.5, the performance of the trained model is the best. The experimental results show that the F1-Score of bounding box and mask region all achieve 92.0%. The image acquisition processes are fully unobservable, without any user preselection, which are a highly heterogenic mix, the selected Mask R-CNN algorithm could also accurately detect mature green tomatoes. The performance of this proposed model in a real greenhouse harvesting environment is also evaluated, thus facilitating the direct application in a tomato harvesting robot.https://www.mdpi.com/1424-8220/21/23/7842Mask R-CNNdetection and segmentationmature green tomatomobile robot |
| spellingShingle | Linlu Zu Yanping Zhao Jiuqin Liu Fei Su Yan Zhang Pingzeng Liu Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach Sensors Mask R-CNN detection and segmentation mature green tomato mobile robot |
| title | Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach |
| title_full | Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach |
| title_fullStr | Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach |
| title_full_unstemmed | Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach |
| title_short | Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach |
| title_sort | detection and segmentation of mature green tomatoes based on mask r cnn with automatic image acquisition approach |
| topic | Mask R-CNN detection and segmentation mature green tomato mobile robot |
| url | https://www.mdpi.com/1424-8220/21/23/7842 |
| work_keys_str_mv | AT linluzu detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach AT yanpingzhao detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach AT jiuqinliu detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach AT feisu detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach AT yanzhang detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach AT pingzengliu detectionandsegmentationofmaturegreentomatoesbasedonmaskrcnnwithautomaticimageacquisitionapproach |