kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies
Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Transactions on Machine Learning Research
2024
|
| _version_ | 1826313757991632896 |
|---|---|
| author | Gui, Z Sun, S Li, R Yuan, J An, Z Roth, K Prabhu, A Torr, P |
| author_facet | Gui, Z Sun, S Li, R Yuan, J An, Z Roth, K Prabhu, A Torr, P |
| author_sort | Gui, Z |
| collection | OXFORD |
| description | Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods. |
| first_indexed | 2024-09-25T04:21:37Z |
| format | Journal article |
| id | oxford-uuid:1ce39d5e-fd78-4d29-95e0-cb167c0254bf |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:21:37Z |
| publishDate | 2024 |
| publisher | Transactions on Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:1ce39d5e-fd78-4d29-95e0-cb167c0254bf2024-08-12T13:40:43ZkNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabulariesJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:1ce39d5e-fd78-4d29-95e0-cb167c0254bfEnglishSymplectic ElementsTransactions on Machine Learning Research2024Gui, ZSun, SLi, RYuan, JAn, ZRoth, KPrabhu, ATorr, PContinual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods. |
| spellingShingle | Gui, Z Sun, S Li, R Yuan, J An, Z Roth, K Prabhu, A Torr, P kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title | kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title_full | kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title_fullStr | kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title_full_unstemmed | kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title_short | kNN-CLIP: retrieval enables training-free segmentation on continually expanding large vocabularies |
| title_sort | knn clip retrieval enables training free segmentation on continually expanding large vocabularies |
| work_keys_str_mv | AT guiz knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT suns knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT lir knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT yuanj knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT anz knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT rothk knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT prabhua knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies AT torrp knnclipretrievalenablestrainingfreesegmentationoncontinuallyexpandinglargevocabularies |