Image retrieval outperforms diffusion models on data augmentation
Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contrib...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Journal of Machine Learning Research
2023
|
| _version_ | 1797112822849601536 |
|---|---|
| author | Burg, MF Wenzel, F Zietlow, D Horn, M Makansi, O Locatello, F Russell, C |
| author_facet | Burg, MF Wenzel, F Zietlow, D Horn, M Makansi, O Locatello, F Russell, C |
| author_sort | Burg, MF |
| collection | OXFORD |
| description | Many approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contribute to downstream classification performance. In particular, it remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential of diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retrieval baseline on simple downstream vision tasks. |
| first_indexed | 2024-03-07T08:20:19Z |
| format | Journal article |
| id | oxford-uuid:85e5ba8f-f3b1-4e86-bf21-ea53df69b9ae |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-04-09T03:54:55Z |
| publishDate | 2023 |
| publisher | Journal of Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:85e5ba8f-f3b1-4e86-bf21-ea53df69b9ae2024-03-07T15:18:18ZImage retrieval outperforms diffusion models on data augmentationJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:85e5ba8f-f3b1-4e86-bf21-ea53df69b9aeEnglishSymplectic ElementsJournal of Machine Learning Research2023Burg, MFWenzel, FZietlow, DHorn, MMakansi, OLocatello, FRussell, CMany approaches have been proposed to use diffusion models to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large datasets, often with noisy annotations, and it remains an open question to which extent these models contribute to downstream classification performance. In particular, it remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. We systematically evaluate a range of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance. Our study explores the potential of diffusion models in generating new training data, and surprisingly finds that these sophisticated models are not yet able to beat a simple and strong image retrieval baseline on simple downstream vision tasks. |
| spellingShingle | Burg, MF Wenzel, F Zietlow, D Horn, M Makansi, O Locatello, F Russell, C Image retrieval outperforms diffusion models on data augmentation |
| title | Image retrieval outperforms diffusion models on data augmentation |
| title_full | Image retrieval outperforms diffusion models on data augmentation |
| title_fullStr | Image retrieval outperforms diffusion models on data augmentation |
| title_full_unstemmed | Image retrieval outperforms diffusion models on data augmentation |
| title_short | Image retrieval outperforms diffusion models on data augmentation |
| title_sort | image retrieval outperforms diffusion models on data augmentation |
| work_keys_str_mv | AT burgmf imageretrievaloutperformsdiffusionmodelsondataaugmentation AT wenzelf imageretrievaloutperformsdiffusionmodelsondataaugmentation AT zietlowd imageretrievaloutperformsdiffusionmodelsondataaugmentation AT hornm imageretrievaloutperformsdiffusionmodelsondataaugmentation AT makansio imageretrievaloutperformsdiffusionmodelsondataaugmentation AT locatellof imageretrievaloutperformsdiffusionmodelsondataaugmentation AT russellc imageretrievaloutperformsdiffusionmodelsondataaugmentation |