Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation
In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360◦ imagery. To tackle these problems, first, we propose the upgraded Transformer for...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2024
|
| _version_ | 1817931564514803712 |
|---|---|
| author | Zhang, J Yang, K Shi, H Reiß, S Peng, K Ma, C Fu, H Torr, P Wang, K Stiefelhagen, R |
| author_facet | Zhang, J Yang, K Shi, H Reiß, S Peng, K Ma, C Fu, H Torr, P Wang, K Stiefelhagen, R |
| author_sort | Zhang, J |
| collection | OXFORD |
| description | In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image
distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360◦ imagery. To tackle these problems, first,
we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch
Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before
or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy
via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic
(PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (SYN2REAL)
adaptation scheme in 360◦ imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of
them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain
adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS. |
| first_indexed | 2024-09-25T04:29:54Z |
| format | Conference item |
| id | oxford-uuid:8dce4c8e-aa32-43bd-8cdb-e605c9777dc2 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-12-09T03:24:02Z |
| publishDate | 2024 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:8dce4c8e-aa32-43bd-8cdb-e605c9777dc22024-11-21T20:03:32ZBehind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentationConference itemhttp://purl.org/coar/resource_type/c_5794uuid:8dce4c8e-aa32-43bd-8cdb-e605c9777dc2EnglishSymplectic ElementsIEEE2024Zhang, JYang, KShi, HReiß, SPeng, KMa, CFu, HTorr, PWang, KStiefelhagen, RIn this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360◦ imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (SYN2REAL) adaptation scheme in 360◦ imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS. |
| spellingShingle | Zhang, J Yang, K Shi, H Reiß, S Peng, K Ma, C Fu, H Torr, P Wang, K Stiefelhagen, R Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title | Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title_full | Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title_fullStr | Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title_full_unstemmed | Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title_short | Behind every domain there is a shift: adapting distortion-aware vision transformers for panoramic semantic segmentation |
| title_sort | behind every domain there is a shift adapting distortion aware vision transformers for panoramic semantic segmentation |
| work_keys_str_mv | AT zhangj behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT yangk behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT shih behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT reißs behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT pengk behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT mac behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT fuh behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT torrp behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT wangk behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation AT stiefelhagenr behindeverydomainthereisashiftadaptingdistortionawarevisiontransformersforpanoramicsemanticsegmentation |