Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms
Robust and precise visual localization over extended periods of time poses a formidable challenge in the current domain of spatial vision. The primary difficulty lies in effectively addressing significant variations in appearance caused by seasonal changes (summer, winter, spring, autumn) and divers...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/23/9580 |
| _version_ | 1797399546604552192 |
|---|---|
| author | Sheng Han Wei Gao Zhanyi Hu |
| author_facet | Sheng Han Wei Gao Zhanyi Hu |
| author_sort | Sheng Han |
| collection | DOAJ |
| description | Robust and precise visual localization over extended periods of time poses a formidable challenge in the current domain of spatial vision. The primary difficulty lies in effectively addressing significant variations in appearance caused by seasonal changes (summer, winter, spring, autumn) and diverse lighting conditions (dawn, day, sunset, night). With the rapid development of related technologies, more and more relevant datasets have emerged, which has also promoted the progress of 6-DOF visual localization in both directions of autonomous vehicles and handheld devices.This manuscript endeavors to rectify the existing limitations of the current public benchmark for long-term visual localization, especially in the part on the autonomous vehicle challenge. Taking into account that autonomous vehicle datasets are primarily captured by multi-camera rigs with fixed extrinsic camera calibration and consist of serialized image sequences, we present several proposed modifications designed to enhance the rationality and comprehensiveness of the evaluation algorithm. We advocate for standardized preprocessing procedures to minimize the possibility of human intervention influencing evaluation results. These procedures involve aligning the positions of multiple cameras on the vehicle with a predetermined canonical reference system, replacing the individual camera positions with uniform vehicle poses, and incorporating sequence information to compensate for any failed localized poses. These steps are crucial in ensuring a just and accurate evaluation of algorithmic performance. Lastly, we introduce a novel indicator to resolve potential ties in the Schulze ranking among submitted methods. The inadequacies highlighted in this study are substantiated through simulations and actual experiments, which unequivocally demonstrate the necessity and effectiveness of our proposed amendments. |
| first_indexed | 2024-03-09T01:41:45Z |
| format | Article |
| id | doaj.art-1913133680954c78b273225e798fb811 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T01:41:45Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-1913133680954c78b273225e798fb8112023-12-08T15:26:29ZengMDPI AGSensors1424-82202023-12-012323958010.3390/s23239580Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization AlgorithmsSheng Han0Wei Gao1Zhanyi Hu2School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, ChinaSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, ChinaSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, ChinaRobust and precise visual localization over extended periods of time poses a formidable challenge in the current domain of spatial vision. The primary difficulty lies in effectively addressing significant variations in appearance caused by seasonal changes (summer, winter, spring, autumn) and diverse lighting conditions (dawn, day, sunset, night). With the rapid development of related technologies, more and more relevant datasets have emerged, which has also promoted the progress of 6-DOF visual localization in both directions of autonomous vehicles and handheld devices.This manuscript endeavors to rectify the existing limitations of the current public benchmark for long-term visual localization, especially in the part on the autonomous vehicle challenge. Taking into account that autonomous vehicle datasets are primarily captured by multi-camera rigs with fixed extrinsic camera calibration and consist of serialized image sequences, we present several proposed modifications designed to enhance the rationality and comprehensiveness of the evaluation algorithm. We advocate for standardized preprocessing procedures to minimize the possibility of human intervention influencing evaluation results. These procedures involve aligning the positions of multiple cameras on the vehicle with a predetermined canonical reference system, replacing the individual camera positions with uniform vehicle poses, and incorporating sequence information to compensate for any failed localized poses. These steps are crucial in ensuring a just and accurate evaluation of algorithmic performance. Lastly, we introduce a novel indicator to resolve potential ties in the Schulze ranking among submitted methods. The inadequacies highlighted in this study are substantiated through simulations and actual experiments, which unequivocally demonstrate the necessity and effectiveness of our proposed amendments.https://www.mdpi.com/1424-8220/23/23/9580visual localizationbenchmark enhancementpose compensationsequential interpolationties resolution |
| spellingShingle | Sheng Han Wei Gao Zhanyi Hu Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms Sensors visual localization benchmark enhancement pose compensation sequential interpolation ties resolution |
| title | Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms |
| title_full | Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms |
| title_fullStr | Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms |
| title_full_unstemmed | Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms |
| title_short | Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms |
| title_sort | why you cannot rank first modifications for benchmarking six degree of freedom visual localization algorithms |
| topic | visual localization benchmark enhancement pose compensation sequential interpolation ties resolution |
| url | https://www.mdpi.com/1424-8220/23/23/9580 |
| work_keys_str_mv | AT shenghan whyyoucannotrankfirstmodificationsforbenchmarkingsixdegreeoffreedomvisuallocalizationalgorithms AT weigao whyyoucannotrankfirstmodificationsforbenchmarkingsixdegreeoffreedomvisuallocalizationalgorithms AT zhanyihu whyyoucannotrankfirstmodificationsforbenchmarkingsixdegreeoffreedomvisuallocalizationalgorithms |