A general-purpose framework for parallel processing of large-scale LiDAR data
Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve l...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2018-01-01
|
| Series: | International Journal of Digital Earth |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/17538947.2016.1269842 |
| _version_ | 1797678652651995136 |
|---|---|
| author | Zhenlong Li Michael E. Hodgson Wenwen Li |
| author_facet | Zhenlong Li Michael E. Hodgson Wenwen Li |
| author_sort | Zhenlong Li |
| collection | DOAJ |
| description | Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework. |
| first_indexed | 2024-03-11T23:03:03Z |
| format | Article |
| id | doaj.art-d97f4b48bf81455bbd2ec0646ce31c0d |
| institution | Directory Open Access Journal |
| issn | 1753-8947 1753-8955 |
| language | English |
| last_indexed | 2024-03-11T23:03:03Z |
| publishDate | 2018-01-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | International Journal of Digital Earth |
| spelling | doaj.art-d97f4b48bf81455bbd2ec0646ce31c0d2023-09-21T14:38:05ZengTaylor & Francis GroupInternational Journal of Digital Earth1753-89471753-89552018-01-01111264710.1080/17538947.2016.12698421269842A general-purpose framework for parallel processing of large-scale LiDAR dataZhenlong Li0Michael E. Hodgson1Wenwen Li2University of South CarolinaUniversity of South CarolinaArizona State UniversityLight detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.http://dx.doi.org/10.1080/17538947.2016.1269842big dataonline geoprocessinghadoop mapreducespatial decompositionlastoolsparallel |
| spellingShingle | Zhenlong Li Michael E. Hodgson Wenwen Li A general-purpose framework for parallel processing of large-scale LiDAR data International Journal of Digital Earth big data online geoprocessing hadoop mapreduce spatial decomposition lastools parallel |
| title | A general-purpose framework for parallel processing of large-scale LiDAR data |
| title_full | A general-purpose framework for parallel processing of large-scale LiDAR data |
| title_fullStr | A general-purpose framework for parallel processing of large-scale LiDAR data |
| title_full_unstemmed | A general-purpose framework for parallel processing of large-scale LiDAR data |
| title_short | A general-purpose framework for parallel processing of large-scale LiDAR data |
| title_sort | general purpose framework for parallel processing of large scale lidar data |
| topic | big data online geoprocessing hadoop mapreduce spatial decomposition lastools parallel |
| url | http://dx.doi.org/10.1080/17538947.2016.1269842 |
| work_keys_str_mv | AT zhenlongli ageneralpurposeframeworkforparallelprocessingoflargescalelidardata AT michaelehodgson ageneralpurposeframeworkforparallelprocessingoflargescalelidardata AT wenwenli ageneralpurposeframeworkforparallelprocessingoflargescalelidardata AT zhenlongli generalpurposeframeworkforparallelprocessingoflargescalelidardata AT michaelehodgson generalpurposeframeworkforparallelprocessingoflargescalelidardata AT wenwenli generalpurposeframeworkforparallelprocessingoflargescalelidardata |