O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments
Automated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | Smart Agricultural Technology |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2772375523001132 |
| _version_ | 1797780019532005376 |
|---|---|
| author | Pengyu Chu Zhaojian Li Kaixiang Zhang Dong Chen Kyle Lammers Renfu Lu |
| author_facet | Pengyu Chu Zhaojian Li Kaixiang Zhang Dong Chen Kyle Lammers Renfu Lu |
| author_sort | Pengyu Chu |
| collection | DOAJ |
| description | Automated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments that involve varying lighting conditions, fruit clustering and foliage/branch occlusions. Apples are often grown in clusters on trees, which may be mis-identified as a single apple and thus causes issues in fruit localization for subsequent robotic harvesting operations. In this paper, we present the development of a novel deep learning-based apple detection framework, called the Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in clustered situations. A comprehensive dataset of RGB images were collected for two varieties of apples under different lighting conditions (overcast, direct lighting, and back lighting) with varying degrees of apple occlusions, and the images were annotated and made available to the public. A novel occlusion-aware network was developed for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations of the developed O2RNet were performed using the collected images, which outperformed 12 other state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection. O2RNet provides an enhanced method for robust detection of clustered apples, which is critical to accurate fruit localization for robotic harvesting. |
| first_indexed | 2024-03-12T23:38:44Z |
| format | Article |
| id | doaj.art-647319b6a34747f5b685aafc02981c16 |
| institution | Directory Open Access Journal |
| issn | 2772-3755 |
| language | English |
| last_indexed | 2024-03-12T23:38:44Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Smart Agricultural Technology |
| spelling | doaj.art-647319b6a34747f5b685aafc02981c162023-07-15T04:29:24ZengElsevierSmart Agricultural Technology2772-37552023-10-015100284O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environmentsPengyu Chu0Zhaojian Li1Kaixiang Zhang2Dong Chen3Kyle Lammers4Renfu Lu5Department of Mechanical Engineering, Michigan State University, East Lansing, MI 48824, USADepartment of Mechanical Engineering, Michigan State University, East Lansing, MI 48824, USA; Corresponding author.Department of Mechanical Engineering, Michigan State University, East Lansing, MI 48824, USADepartment of Mechanical Engineering, Michigan State University, East Lansing, MI 48824, USADepartment of Mechanical Engineering, Michigan State University, East Lansing, MI 48824, USADepartment of Agriculture (USDA) Agricultural Research Service (ARS), East Lansing, MI 48824, USAAutomated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments that involve varying lighting conditions, fruit clustering and foliage/branch occlusions. Apples are often grown in clusters on trees, which may be mis-identified as a single apple and thus causes issues in fruit localization for subsequent robotic harvesting operations. In this paper, we present the development of a novel deep learning-based apple detection framework, called the Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in clustered situations. A comprehensive dataset of RGB images were collected for two varieties of apples under different lighting conditions (overcast, direct lighting, and back lighting) with varying degrees of apple occlusions, and the images were annotated and made available to the public. A novel occlusion-aware network was developed for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations of the developed O2RNet were performed using the collected images, which outperformed 12 other state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection. O2RNet provides an enhanced method for robust detection of clustered apples, which is critical to accurate fruit localization for robotic harvesting.http://www.sciencedirect.com/science/article/pii/S2772375523001132Computer visionApple detectionFruit harvestingOcclusion-aware detectionTransfer learning |
| spellingShingle | Pengyu Chu Zhaojian Li Kaixiang Zhang Dong Chen Kyle Lammers Renfu Lu O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments Smart Agricultural Technology Computer vision Apple detection Fruit harvesting Occlusion-aware detection Transfer learning |
| title | O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments |
| title_full | O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments |
| title_fullStr | O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments |
| title_full_unstemmed | O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments |
| title_short | O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments |
| title_sort | o2rnet occluder occludee relational network for robust apple detection in clustered orchard environments |
| topic | Computer vision Apple detection Fruit harvesting Occlusion-aware detection Transfer learning |
| url | http://www.sciencedirect.com/science/article/pii/S2772375523001132 |
| work_keys_str_mv | AT pengyuchu o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments AT zhaojianli o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments AT kaixiangzhang o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments AT dongchen o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments AT kylelammers o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments AT renfulu o2rnetoccluderoccludeerelationalnetworkforrobustappledetectioninclusteredorchardenvironments |