CISA: Context Substitution for Image Semantics Augmentation
Large datasets catalyze the rapid expansion of deep learning and computer vision. At the same time, in many domains, there is a lack of training data, which may become an obstacle for the practical application of deep computer vision models. To overcome this problem, it is popular to apply image aug...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/11/8/1818 |
| _version_ | 1797604502468034560 |
|---|---|
| author | Sergey Nesteruk Ilya Zherebtsov Svetlana Illarionova Dmitrii Shadrin Andrey Somov Sergey V. Bezzateev Tatiana Yelina Vladimir Denisenko Ivan Oseledets |
| author_facet | Sergey Nesteruk Ilya Zherebtsov Svetlana Illarionova Dmitrii Shadrin Andrey Somov Sergey V. Bezzateev Tatiana Yelina Vladimir Denisenko Ivan Oseledets |
| author_sort | Sergey Nesteruk |
| collection | DOAJ |
| description | Large datasets catalyze the rapid expansion of deep learning and computer vision. At the same time, in many domains, there is a lack of training data, which may become an obstacle for the practical application of deep computer vision models. To overcome this problem, it is popular to apply image augmentation. When a dataset contains instance segmentation masks, it is possible to apply instance-level augmentation. It operates by cutting an instance from the original image and pasting to new backgrounds. This article challenges a dataset with the same objects present in various domains. We introduce the Context Substitution for Image Semantics Augmentation framework (CISA), which is focused on choosing good background images. We compare several ways to find backgrounds that match the context of the test set, including Contrastive Language–Image Pre-Training (CLIP) image retrieval and diffusion image generation. We prove that our augmentation method is effective for classification, segmentation, and object detection with different dataset complexity and different model types. The average percentage increase in accuracy across all the tasks on a fruits and vegetables recognition dataset is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>4.95</mn><mo>%</mo></mrow></semantics></math></inline-formula>. Moreover, we show that the Fréchet Inception Distance (FID) metrics has a strong correlation with model accuracy, and it can help to choose better backgrounds without model training. The average negative correlation between model accuracy and the FID between the augmented and test datasets is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>0.55</mn></mrow></semantics></math></inline-formula> in our experiments. |
| first_indexed | 2024-03-11T04:47:12Z |
| format | Article |
| id | doaj.art-f6c573f93e0d418ab0eaa39bda13493e |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-11T04:47:12Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-f6c573f93e0d418ab0eaa39bda13493e2023-11-17T20:16:57ZengMDPI AGMathematics2227-73902023-04-01118181810.3390/math11081818CISA: Context Substitution for Image Semantics AugmentationSergey Nesteruk0Ilya Zherebtsov1Svetlana Illarionova2Dmitrii Shadrin3Andrey Somov4Sergey V. Bezzateev5Tatiana Yelina6Vladimir Denisenko7Ivan Oseledets8Skolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, RussiaVoronezh State University of Engineering Technology (VSUET), 394036 Voronezh, RussiaSkolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, RussiaSkolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, RussiaSkolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, RussiaSaint-Petrsburg State University of Aerospace Instrumentation (SUAI), 190000 Saint Petersburg, RussiaSaint-Petrsburg State University of Aerospace Instrumentation (SUAI), 190000 Saint Petersburg, RussiaVoronezh State University of Engineering Technology (VSUET), 394036 Voronezh, RussiaSkolkovo Institute of Science and Technology (Skoltech), 121205 Moscow, RussiaLarge datasets catalyze the rapid expansion of deep learning and computer vision. At the same time, in many domains, there is a lack of training data, which may become an obstacle for the practical application of deep computer vision models. To overcome this problem, it is popular to apply image augmentation. When a dataset contains instance segmentation masks, it is possible to apply instance-level augmentation. It operates by cutting an instance from the original image and pasting to new backgrounds. This article challenges a dataset with the same objects present in various domains. We introduce the Context Substitution for Image Semantics Augmentation framework (CISA), which is focused on choosing good background images. We compare several ways to find backgrounds that match the context of the test set, including Contrastive Language–Image Pre-Training (CLIP) image retrieval and diffusion image generation. We prove that our augmentation method is effective for classification, segmentation, and object detection with different dataset complexity and different model types. The average percentage increase in accuracy across all the tasks on a fruits and vegetables recognition dataset is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>4.95</mn><mo>%</mo></mrow></semantics></math></inline-formula>. Moreover, we show that the Fréchet Inception Distance (FID) metrics has a strong correlation with model accuracy, and it can help to choose better backgrounds without model training. The average negative correlation between model accuracy and the FID between the augmented and test datasets is <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>0.55</mn></mrow></semantics></math></inline-formula> in our experiments.https://www.mdpi.com/2227-7390/11/8/1818image augmentationcomputer visiondata collectionimage retrievalimage generationfew-shot learning |
| spellingShingle | Sergey Nesteruk Ilya Zherebtsov Svetlana Illarionova Dmitrii Shadrin Andrey Somov Sergey V. Bezzateev Tatiana Yelina Vladimir Denisenko Ivan Oseledets CISA: Context Substitution for Image Semantics Augmentation Mathematics image augmentation computer vision data collection image retrieval image generation few-shot learning |
| title | CISA: Context Substitution for Image Semantics Augmentation |
| title_full | CISA: Context Substitution for Image Semantics Augmentation |
| title_fullStr | CISA: Context Substitution for Image Semantics Augmentation |
| title_full_unstemmed | CISA: Context Substitution for Image Semantics Augmentation |
| title_short | CISA: Context Substitution for Image Semantics Augmentation |
| title_sort | cisa context substitution for image semantics augmentation |
| topic | image augmentation computer vision data collection image retrieval image generation few-shot learning |
| url | https://www.mdpi.com/2227-7390/11/8/1818 |
| work_keys_str_mv | AT sergeynesteruk cisacontextsubstitutionforimagesemanticsaugmentation AT ilyazherebtsov cisacontextsubstitutionforimagesemanticsaugmentation AT svetlanaillarionova cisacontextsubstitutionforimagesemanticsaugmentation AT dmitriishadrin cisacontextsubstitutionforimagesemanticsaugmentation AT andreysomov cisacontextsubstitutionforimagesemanticsaugmentation AT sergeyvbezzateev cisacontextsubstitutionforimagesemanticsaugmentation AT tatianayelina cisacontextsubstitutionforimagesemanticsaugmentation AT vladimirdenisenko cisacontextsubstitutionforimagesemanticsaugmentation AT ivanoseledets cisacontextsubstitutionforimagesemanticsaugmentation |