Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data
Precise single tree delineation allows for a more reliable determination of essential parameters such as tree species, height and vitality. Methods of instance segmentation are powerful neural networks for detecting and segmenting single objects and have the potential to push the accuracy of tree se...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-04-01
|
| Series: | ISPRS Open Journal of Photogrammetry and Remote Sensing |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S266739322300008X |
| _version_ | 1797828109200785408 |
|---|---|
| author | S. Dersch A. Schöttl P. Krzystek M. Heurich |
| author_facet | S. Dersch A. Schöttl P. Krzystek M. Heurich |
| author_sort | S. Dersch |
| collection | DOAJ |
| description | Precise single tree delineation allows for a more reliable determination of essential parameters such as tree species, height and vitality. Methods of instance segmentation are powerful neural networks for detecting and segmenting single objects and have the potential to push the accuracy of tree segmentation methods to a new level. In this study, two instance segmentation methods, Mask R–CNN and DETR, were applied to precisely delineate single tree crowns using multispectral images and images generated from UAV lidar data. The study area was in Bavaria, 35 km north of Munich (Germany), comprising a mixed forest stand of around 7 ha characterised mainly by Norway spruce (Picea abies) and large groups of European beeches (Fagus sylvatica) with 181–236 trees per ha. The data set, consisting of multispectral images and lidar data, was acquired using a Micasense RedEdge-MX dual camera system and a Riegl miniVUX-1UAV lidar scanner, both mounted on a hexacopter (DJI Matrice 600 Pro). At an altitude of approximately 85 m, two flight missions were conducted at an airspeed of 5 m/s, leading to a ground resolution of 5 cm and a lidar point density of 560 points/m2. In total, 1408 trees were marked by visual interpretation of the remote sensing data for training and validating the classifiers. Additionally, 125 trees were surveyed by tacheometric means used to test the optimized neural networks. The evaluations showed that segmentation using only multispectral imagery performed slightly better than with images generated from lidar data. In terms of F1 score, Mask R–CNN with color infrared (CIR) images achieved 92% in coniferous, 85% in deciduous and 83% in mixed stands. Compared to the images generated by lidar data, these scores are the same for coniferous and slightly worse for deciduous and mixed plots, by 4% and 2%, respectively. DETR with CIR images achieved 90% in coniferous, 81% in deciduous and 84% in mixed stands. These scores were 2%, 1%, and 2% worse, respectively, compared to the lidar data images in the same test areas. Interestingly, four conventional segmentation methods performed significantly worse than CIR-based and lidar-based instance segmentations. Additionally, the results revealed that tree crowns were more accurately segmented by instance segmentation. All in all, the results highlight the practical potential of the two deep learning-based tree segmentation methods, especially in comparison to baseline methods. |
| first_indexed | 2024-04-09T12:59:41Z |
| format | Article |
| id | doaj.art-3c721d92ca8f4b3e97fcc5a811afc5b4 |
| institution | Directory Open Access Journal |
| issn | 2667-3932 |
| language | English |
| last_indexed | 2024-04-09T12:59:41Z |
| publishDate | 2023-04-01 |
| publisher | Elsevier |
| record_format | Article |
| series | ISPRS Open Journal of Photogrammetry and Remote Sensing |
| spelling | doaj.art-3c721d92ca8f4b3e97fcc5a811afc5b42023-05-13T04:25:59ZengElsevierISPRS Open Journal of Photogrammetry and Remote Sensing2667-39322023-04-018100037Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar dataS. Dersch0A. Schöttl1P. Krzystek2M. Heurich3Dept. of Geoinformatics, Munich University of Applied Sciences, 80333, Munich, Germany; Faculty of Environment and Natural Resources, University of Freiburg, Germany; Institute for Applications of Machine Learning and Intelligent Systems, Munich University of Applied Sciences, 80335, Munich, Germany; Corresponding author. Dept. of Geoinformatics, Munich University of Applied Sciences, 80333, Munich, Germany.Dept. of Electrical Engineering and Information Technology, Munich University of Applied Sciences, 80335, Munich, Germany; Institute for Applications of Machine Learning and Intelligent Systems, Munich University of Applied Sciences, 80335, Munich, GermanyDept. of Geoinformatics, Munich University of Applied Sciences, 80333, Munich, Germany; Institute for Applications of Machine Learning and Intelligent Systems, Munich University of Applied Sciences, 80335, Munich, GermanyFaculty of Environment and Natural Resources, University of Freiburg, Germany; Bavarian Forest National Park, Dept. of Visitor Management and National Park Monitoring, 94481, Grafenau, Germany; Institute of Forestry and Wildlife Management, Inland Norway University of Applied Science, NO-2480, Koppang, NorwayPrecise single tree delineation allows for a more reliable determination of essential parameters such as tree species, height and vitality. Methods of instance segmentation are powerful neural networks for detecting and segmenting single objects and have the potential to push the accuracy of tree segmentation methods to a new level. In this study, two instance segmentation methods, Mask R–CNN and DETR, were applied to precisely delineate single tree crowns using multispectral images and images generated from UAV lidar data. The study area was in Bavaria, 35 km north of Munich (Germany), comprising a mixed forest stand of around 7 ha characterised mainly by Norway spruce (Picea abies) and large groups of European beeches (Fagus sylvatica) with 181–236 trees per ha. The data set, consisting of multispectral images and lidar data, was acquired using a Micasense RedEdge-MX dual camera system and a Riegl miniVUX-1UAV lidar scanner, both mounted on a hexacopter (DJI Matrice 600 Pro). At an altitude of approximately 85 m, two flight missions were conducted at an airspeed of 5 m/s, leading to a ground resolution of 5 cm and a lidar point density of 560 points/m2. In total, 1408 trees were marked by visual interpretation of the remote sensing data for training and validating the classifiers. Additionally, 125 trees were surveyed by tacheometric means used to test the optimized neural networks. The evaluations showed that segmentation using only multispectral imagery performed slightly better than with images generated from lidar data. In terms of F1 score, Mask R–CNN with color infrared (CIR) images achieved 92% in coniferous, 85% in deciduous and 83% in mixed stands. Compared to the images generated by lidar data, these scores are the same for coniferous and slightly worse for deciduous and mixed plots, by 4% and 2%, respectively. DETR with CIR images achieved 90% in coniferous, 81% in deciduous and 84% in mixed stands. These scores were 2%, 1%, and 2% worse, respectively, compared to the lidar data images in the same test areas. Interestingly, four conventional segmentation methods performed significantly worse than CIR-based and lidar-based instance segmentations. Additionally, the results revealed that tree crowns were more accurately segmented by instance segmentation. All in all, the results highlight the practical potential of the two deep learning-based tree segmentation methods, especially in comparison to baseline methods.http://www.sciencedirect.com/science/article/pii/S266739322300008XSingle tree delineationInstance segmentationMultispectral imageryLidar |
| spellingShingle | S. Dersch A. Schöttl P. Krzystek M. Heurich Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data ISPRS Open Journal of Photogrammetry and Remote Sensing Single tree delineation Instance segmentation Multispectral imagery Lidar |
| title | Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data |
| title_full | Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data |
| title_fullStr | Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data |
| title_full_unstemmed | Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data |
| title_short | Towards complete tree crown delineation by instance segmentation with Mask R–CNN and DETR using UAV-based multispectral imagery and lidar data |
| title_sort | towards complete tree crown delineation by instance segmentation with mask r cnn and detr using uav based multispectral imagery and lidar data |
| topic | Single tree delineation Instance segmentation Multispectral imagery Lidar |
| url | http://www.sciencedirect.com/science/article/pii/S266739322300008X |
| work_keys_str_mv | AT sdersch towardscompletetreecrowndelineationbyinstancesegmentationwithmaskrcnnanddetrusinguavbasedmultispectralimageryandlidardata AT aschottl towardscompletetreecrowndelineationbyinstancesegmentationwithmaskrcnnanddetrusinguavbasedmultispectralimageryandlidardata AT pkrzystek towardscompletetreecrowndelineationbyinstancesegmentationwithmaskrcnnanddetrusinguavbasedmultispectralimageryandlidardata AT mheurich towardscompletetreecrowndelineationbyinstancesegmentationwithmaskrcnnanddetrusinguavbasedmultispectralimageryandlidardata |