RobustMat: neural diffusion for street landmark patch matching under challenging environments
For autonomous vehicles (AVs), visual perception techniques based on sensors like cameras play crucial roles in information acquisition and processing. In various computer perception tasks for AVs, it may be helpful to match landmark patches taken by an onboard camera with other landmark patches cap...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/173506 |
| _version_ | 1824456578924806144 |
|---|---|
| author | She, Rui Kang, Qiyu Wang, Sijie Yang, Yuan-Rui Zhao, Kai Song, Yang Tay, Wee Peng |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering She, Rui Kang, Qiyu Wang, Sijie Yang, Yuan-Rui Zhao, Kai Song, Yang Tay, Wee Peng |
| author_sort | She, Rui |
| collection | NTU |
| description | For autonomous vehicles (AVs), visual perception techniques based on sensors like cameras play crucial roles in information acquisition and processing. In various computer perception tasks for AVs, it may be helpful to match landmark patches taken by an onboard camera with other landmark patches captured at a different time or saved in a street scene image database. To perform matching under challenging driving environments caused by changing seasons, weather, and illumination, we utilize the spatial neighborhood information of each patch. We propose an approach, named RobustMat, which derives its robustness to perturbations from neural differential equations. A convolutional neural ODE diffusion module is used to learn the feature representation for the landmark patches. A graph neural PDE diffusion module then aggregates information from neighboring landmark patches in the street scene. Finally, feature similarity learning outputs the final matching score. Our approach is evaluated on several street scene datasets and demonstrated to achieve state-of-the-art matching results under environmental perturbations. |
| first_indexed | 2025-02-19T03:56:20Z |
| format | Journal Article |
| id | ntu-10356/173506 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:56:20Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1735062024-02-07T07:31:09Z RobustMat: neural diffusion for street landmark patch matching under challenging environments She, Rui Kang, Qiyu Wang, Sijie Yang, Yuan-Rui Zhao, Kai Song, Yang Tay, Wee Peng School of Electrical and Electronic Engineering Continental-NTU Corporate Lab Engineering Image Matching Neural Diffusion For autonomous vehicles (AVs), visual perception techniques based on sensors like cameras play crucial roles in information acquisition and processing. In various computer perception tasks for AVs, it may be helpful to match landmark patches taken by an onboard camera with other landmark patches captured at a different time or saved in a street scene image database. To perform matching under challenging driving environments caused by changing seasons, weather, and illumination, we utilize the spatial neighborhood information of each patch. We propose an approach, named RobustMat, which derives its robustness to perturbations from neural differential equations. A convolutional neural ODE diffusion module is used to learn the feature representation for the landmark patches. A graph neural PDE diffusion module then aggregates information from neighboring landmark patches in the street scene. Finally, feature similarity learning outputs the final matching score. Our approach is evaluated on several street scene datasets and demonstrated to achieve state-of-the-art matching results under environmental perturbations. Agency for Science, Technology and Research (A*STAR) Ministry of Education (MOE) This work was supported by the A*STAR under its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund–Pre Positioning (IAF-PP) under Grant A19D6a0053 and in part by the Singapore Ministry of Education Academic Research Fund Tier 2 Grant under Grant MOE-T2EP20220-0002. 2024-02-07T07:31:09Z 2024-02-07T07:31:09Z 2023 Journal Article She, R., Kang, Q., Wang, S., Yang, Y., Zhao, K., Song, Y. & Tay, W. P. (2023). RobustMat: neural diffusion for street landmark patch matching under challenging environments. IEEE Transactions On Image Processing, 32, 5550-5563. https://dx.doi.org/10.1109/TIP.2023.3318963 1057-7149 https://hdl.handle.net/10356/173506 10.1109/TIP.2023.3318963 37773901 2-s2.0-85173386348 32 5550 5563 en A19D6a0053 MOE-T2EP20220-0002 IEEE Transactions on Image Processing © 2023 IEEE. All rights reserved. |
| spellingShingle | Engineering Image Matching Neural Diffusion She, Rui Kang, Qiyu Wang, Sijie Yang, Yuan-Rui Zhao, Kai Song, Yang Tay, Wee Peng RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title | RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title_full | RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title_fullStr | RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title_full_unstemmed | RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title_short | RobustMat: neural diffusion for street landmark patch matching under challenging environments |
| title_sort | robustmat neural diffusion for street landmark patch matching under challenging environments |
| topic | Engineering Image Matching Neural Diffusion |
| url | https://hdl.handle.net/10356/173506 |
| work_keys_str_mv | AT sherui robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT kangqiyu robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT wangsijie robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT yangyuanrui robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT zhaokai robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT songyang robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments AT tayweepeng robustmatneuraldiffusionforstreetlandmarkpatchmatchingunderchallengingenvironments |