Estimating depth from RGB images using deep-learning for robotic applications in apple orchards
Vision-enabled robotic approaches for apple orchard management have been widely studied in recent years. It is essential for the vision-system to capture the depth information of the canopies for improved understanding of the geometric relations between objects in the orchard environment, which is e...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-12-01
|
| Series: | Smart Agricultural Technology |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2772375523001740 |
| _version_ | 1797390375691747328 |
|---|---|
| author | L.G. Divyanth Divya Rathore Piranav Senthilkumar Prakhar Patidar Xin Zhang Manoj Karkee Rajendra Machavaram Peeyush Soni |
| author_facet | L.G. Divyanth Divya Rathore Piranav Senthilkumar Prakhar Patidar Xin Zhang Manoj Karkee Rajendra Machavaram Peeyush Soni |
| author_sort | L.G. Divyanth |
| collection | DOAJ |
| description | Vision-enabled robotic approaches for apple orchard management have been widely studied in recent years. It is essential for the vision-system to capture the depth information of the canopies for improved understanding of the geometric relations between objects in the orchard environment, which is essential for safe and efficient operations of robots. Unfortunately, depth-enabled sensors are more expensive and less ubiquitous compared to standard RGB cameras, thus limiting the accessibility of depth cues. This study demonstrates that a data-driven approach using a conditional generative adversarial network (cGAN), known as Pix2Pix can estimate depth from RGB images of orchards acquired from a monocular camera. The Pix2Pix network was modified to generate a depth channel when a standard RGB image was given as input. The network was trained and tested for their efficacy using images acquired from two different apple cultivation systems and camera models. The results demonstrated that the model can generate depth estimates comparable to the actual depth channel with a root-mean-squared error (RMSE) of 1.83 cm (corresponding to a relative error of 3.5%). Moreover, a high structural similarity measure index (> 0.55) and commensurate textural features were observed between the actual depth image and the predicted depth image. The results showed that the use of the Pix2Pix model for producing rational depth maps of fruit orchards with monocular cameras is a viable alternative to the use of relatively more expensive RGB-D sensors for obtaining depth information. |
| first_indexed | 2024-03-08T23:09:50Z |
| format | Article |
| id | doaj.art-ccfbdefb58ad46e29fef07018020982f |
| institution | Directory Open Access Journal |
| issn | 2772-3755 |
| language | English |
| last_indexed | 2024-03-08T23:09:50Z |
| publishDate | 2023-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Smart Agricultural Technology |
| spelling | doaj.art-ccfbdefb58ad46e29fef07018020982f2023-12-15T07:27:12ZengElsevierSmart Agricultural Technology2772-37552023-12-016100345Estimating depth from RGB images using deep-learning for robotic applications in apple orchardsL.G. Divyanth0Divya Rathore1Piranav Senthilkumar2Prakhar Patidar3Xin Zhang4Manoj Karkee5Rajendra Machavaram6Peeyush Soni7Agricultural and Food Engineering Department, Indian Institute of Technology Kharagpur, Kharagpur 721302 West Bengal, India; Center for Precision and Automated Agricultural Systems, Department of Biological Systems Engineering, Washington State University, Prosser, WA 99350, USAAgricultural and Food Engineering Department, Indian Institute of Technology Kharagpur, Kharagpur 721302 West Bengal, IndiaCentre for Excellence in Artificial Intelligence, Indian Institute of Technology Kharagpur, Kharagpur 721302, West Bengal, IndiaAgricultural and Food Engineering Department, Indian Institute of Technology Kharagpur, Kharagpur 721302 West Bengal, IndiaDepartment of Agricultural and Biological Engineering, Mississippi State University, Mississippi State, MS 39762, USACenter for Precision and Automated Agricultural Systems, Department of Biological Systems Engineering, Washington State University, Prosser, WA 99350, USAAgricultural and Food Engineering Department, Indian Institute of Technology Kharagpur, Kharagpur 721302 West Bengal, IndiaAgricultural and Food Engineering Department, Indian Institute of Technology Kharagpur, Kharagpur 721302 West Bengal, India; Corresponding author.Vision-enabled robotic approaches for apple orchard management have been widely studied in recent years. It is essential for the vision-system to capture the depth information of the canopies for improved understanding of the geometric relations between objects in the orchard environment, which is essential for safe and efficient operations of robots. Unfortunately, depth-enabled sensors are more expensive and less ubiquitous compared to standard RGB cameras, thus limiting the accessibility of depth cues. This study demonstrates that a data-driven approach using a conditional generative adversarial network (cGAN), known as Pix2Pix can estimate depth from RGB images of orchards acquired from a monocular camera. The Pix2Pix network was modified to generate a depth channel when a standard RGB image was given as input. The network was trained and tested for their efficacy using images acquired from two different apple cultivation systems and camera models. The results demonstrated that the model can generate depth estimates comparable to the actual depth channel with a root-mean-squared error (RMSE) of 1.83 cm (corresponding to a relative error of 3.5%). Moreover, a high structural similarity measure index (> 0.55) and commensurate textural features were observed between the actual depth image and the predicted depth image. The results showed that the use of the Pix2Pix model for producing rational depth maps of fruit orchards with monocular cameras is a viable alternative to the use of relatively more expensive RGB-D sensors for obtaining depth information.http://www.sciencedirect.com/science/article/pii/S2772375523001740Generative adversarial networksPix2PixRGB-DMachine visionAgricultural robots |
| spellingShingle | L.G. Divyanth Divya Rathore Piranav Senthilkumar Prakhar Patidar Xin Zhang Manoj Karkee Rajendra Machavaram Peeyush Soni Estimating depth from RGB images using deep-learning for robotic applications in apple orchards Smart Agricultural Technology Generative adversarial networks Pix2Pix RGB-D Machine vision Agricultural robots |
| title | Estimating depth from RGB images using deep-learning for robotic applications in apple orchards |
| title_full | Estimating depth from RGB images using deep-learning for robotic applications in apple orchards |
| title_fullStr | Estimating depth from RGB images using deep-learning for robotic applications in apple orchards |
| title_full_unstemmed | Estimating depth from RGB images using deep-learning for robotic applications in apple orchards |
| title_short | Estimating depth from RGB images using deep-learning for robotic applications in apple orchards |
| title_sort | estimating depth from rgb images using deep learning for robotic applications in apple orchards |
| topic | Generative adversarial networks Pix2Pix RGB-D Machine vision Agricultural robots |
| url | http://www.sciencedirect.com/science/article/pii/S2772375523001740 |
| work_keys_str_mv | AT lgdivyanth estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT divyarathore estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT piranavsenthilkumar estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT prakharpatidar estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT xinzhang estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT manojkarkee estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT rajendramachavaram estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards AT peeyushsoni estimatingdepthfromrgbimagesusingdeeplearningforroboticapplicationsinappleorchards |