Learning universal semantic correspondences with no supervision and automatic data curation
We study the problem of learning semantic image correspondences without manual supervision. Previous works that tackled this problem rely on manually curated image pairs and learn benchmark-specific correspondences. Instead, we present a new method that learns universal correspondences once, from a...
| Main Authors: | , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2023
|
| _version_ | 1797112256884899840 |
|---|---|
| author | Shtedritski, A Vedaldi, A Rupprecth, C |
| author_facet | Shtedritski, A Vedaldi, A Rupprecth, C |
| author_sort | Shtedritski, A |
| collection | OXFORD |
| description | We study the problem of learning semantic image correspondences without manual supervision. Previous works that tackled this problem rely on manually curated image pairs and learn benchmark-specific correspondences. Instead, we present a new method that learns universal correspondences once, from a large image dataset, and without using any manual curation. Despite their generality and despite using less supervision, our universal correspondences still outperform prior works, unsupervised and weakly supervised, in most benchmarks. Our approach starts from local features extracted by an unsupervised vision transformer, which obtain good semantic but poor geometric matching accuracy. It then learns a Transformer Adapter which improves the geometric accuracy of the features, as well as their compatibility between pairs of different images. The method combines semantic similarity with geometric stability obtained via cycle consistency and supervision via synthetic transformations. We use these features to also select pairs of matching images for training the unsupervised correspondences. |
| first_indexed | 2024-03-07T08:23:11Z |
| format | Conference item |
| id | oxford-uuid:67084786-ba38-4d80-afff-9e4f8a69d157 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T08:23:11Z |
| publishDate | 2023 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:67084786-ba38-4d80-afff-9e4f8a69d1572024-02-05T12:13:47ZLearning universal semantic correspondences with no supervision and automatic data curationConference itemhttp://purl.org/coar/resource_type/c_5794uuid:67084786-ba38-4d80-afff-9e4f8a69d157EnglishSymplectic ElementsIEEE2023Shtedritski, AVedaldi, ARupprecth, CWe study the problem of learning semantic image correspondences without manual supervision. Previous works that tackled this problem rely on manually curated image pairs and learn benchmark-specific correspondences. Instead, we present a new method that learns universal correspondences once, from a large image dataset, and without using any manual curation. Despite their generality and despite using less supervision, our universal correspondences still outperform prior works, unsupervised and weakly supervised, in most benchmarks. Our approach starts from local features extracted by an unsupervised vision transformer, which obtain good semantic but poor geometric matching accuracy. It then learns a Transformer Adapter which improves the geometric accuracy of the features, as well as their compatibility between pairs of different images. The method combines semantic similarity with geometric stability obtained via cycle consistency and supervision via synthetic transformations. We use these features to also select pairs of matching images for training the unsupervised correspondences. |
| spellingShingle | Shtedritski, A Vedaldi, A Rupprecth, C Learning universal semantic correspondences with no supervision and automatic data curation |
| title | Learning universal semantic correspondences with no supervision and automatic data curation |
| title_full | Learning universal semantic correspondences with no supervision and automatic data curation |
| title_fullStr | Learning universal semantic correspondences with no supervision and automatic data curation |
| title_full_unstemmed | Learning universal semantic correspondences with no supervision and automatic data curation |
| title_short | Learning universal semantic correspondences with no supervision and automatic data curation |
| title_sort | learning universal semantic correspondences with no supervision and automatic data curation |
| work_keys_str_mv | AT shtedritskia learninguniversalsemanticcorrespondenceswithnosupervisionandautomaticdatacuration AT vedaldia learninguniversalsemanticcorrespondenceswithnosupervisionandautomaticdatacuration AT rupprecthc learninguniversalsemanticcorrespondenceswithnosupervisionandautomaticdatacuration |