Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review
Perception algorithms are essential for autonomous or semi-autonomous vehicles to perceive the semantics of their surroundings, including object detection, panoptic segmentation, and tracking. Decision-making in case of safety-critical situations, like autonomous emergency braking and collision avoi...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10271280/ |
| _version_ | 1827795223078502400 |
|---|---|
| author | Chiranjeevi Karri Jose Machado da Silva Miguel Velhote Correia |
| author_facet | Chiranjeevi Karri Jose Machado da Silva Miguel Velhote Correia |
| author_sort | Chiranjeevi Karri |
| collection | DOAJ |
| description | Perception algorithms are essential for autonomous or semi-autonomous vehicles to perceive the semantics of their surroundings, including object detection, panoptic segmentation, and tracking. Decision-making in case of safety-critical situations, like autonomous emergency braking and collision avoidance, relies on the outputs of these algorithms. This makes it essential to correctly assess such perception systems before their deployment and to monitor their performance when in use. It is difficult to test and validate these systems, particularly at runtime, due to the high-level and complex representations of their outputs. This paper presents an overview of different existing metrics used for the evaluation of LiDAR-based perception systems, emphasizing particularly object detection and tracking algorithms due to their importance in the final perception outcome. Along with generally used metrics, we also discuss the impact of Planning KL-Divergence (PKL), Timed Quality Temporal Logic (TQTL), and Spatio-temporal Quality Logic (STQL) metrics on object detection algorithms. In the case of panoptic segmentation, Panoptic Quality (PQ) and Parsing Covering (PC) metrics are analysed resorting to some pretrained models. Finally, it addresses the application of diverse metrics to evaluate different pretrained models with the respective perception algorithms on publicly available datasets. Besides the identification of the various metrics being proposed, their performance and influence on models are also assessed after conducting new tests or reproducing the experimental results of the reference under consideration. |
| first_indexed | 2024-03-11T18:45:38Z |
| format | Article |
| id | doaj.art-5dc8f70a6a7f4990a50a85f5375f343e |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T18:45:38Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-5dc8f70a6a7f4990a50a85f5375f343e2023-10-11T23:00:27ZengIEEEIEEE Access2169-35362023-01-011110914210916810.1109/ACCESS.2023.332191210271280Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature ReviewChiranjeevi Karri0https://orcid.org/0000-0003-4102-4108Jose Machado da Silva1https://orcid.org/0000-0002-9160-9158Miguel Velhote Correia2https://orcid.org/0000-0001-6065-9358Department of Electrical and Computer Engineering, Faculty of Engineering—University of Porto (FEUP), Porto, PortugalDepartment of Electrical and Computer Engineering, Faculty of Engineering—University of Porto (FEUP), Porto, PortugalDepartment of Electrical and Computer Engineering, Faculty of Engineering—University of Porto (FEUP), Porto, PortugalPerception algorithms are essential for autonomous or semi-autonomous vehicles to perceive the semantics of their surroundings, including object detection, panoptic segmentation, and tracking. Decision-making in case of safety-critical situations, like autonomous emergency braking and collision avoidance, relies on the outputs of these algorithms. This makes it essential to correctly assess such perception systems before their deployment and to monitor their performance when in use. It is difficult to test and validate these systems, particularly at runtime, due to the high-level and complex representations of their outputs. This paper presents an overview of different existing metrics used for the evaluation of LiDAR-based perception systems, emphasizing particularly object detection and tracking algorithms due to their importance in the final perception outcome. Along with generally used metrics, we also discuss the impact of Planning KL-Divergence (PKL), Timed Quality Temporal Logic (TQTL), and Spatio-temporal Quality Logic (STQL) metrics on object detection algorithms. In the case of panoptic segmentation, Panoptic Quality (PQ) and Parsing Covering (PC) metrics are analysed resorting to some pretrained models. Finally, it addresses the application of diverse metrics to evaluate different pretrained models with the respective perception algorithms on publicly available datasets. Besides the identification of the various metrics being proposed, their performance and influence on models are also assessed after conducting new tests or reproducing the experimental results of the reference under consideration.https://ieeexplore.ieee.org/document/10271280/Perception algorithmsmetricsdeep learningobject detectionpanoptic segmentationautonomous driving |
| spellingShingle | Chiranjeevi Karri Jose Machado da Silva Miguel Velhote Correia Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review IEEE Access Perception algorithms metrics deep learning object detection panoptic segmentation autonomous driving |
| title | Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review |
| title_full | Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review |
| title_fullStr | Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review |
| title_full_unstemmed | Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review |
| title_short | Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review |
| title_sort | key indicators to assess the performance of lidar based perception algorithms a literature review |
| topic | Perception algorithms metrics deep learning object detection panoptic segmentation autonomous driving |
| url | https://ieeexplore.ieee.org/document/10271280/ |
| work_keys_str_mv | AT chiranjeevikarri keyindicatorstoassesstheperformanceoflidarbasedperceptionalgorithmsaliteraturereview AT josemachadodasilva keyindicatorstoassesstheperformanceoflidarbasedperceptionalgorithmsaliteraturereview AT miguelvelhotecorreia keyindicatorstoassesstheperformanceoflidarbasedperceptionalgorithmsaliteraturereview |