SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM)
Neural Style Transfer (NST) is a popular technique of computer vision where the content of an image is blended with the style of another, which results in a fused image with certain properties of both original images. This approach has practical applications in various domains and has garnered signi...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10250775/ |
| _version_ | 1827813812184547328 |
|---|---|
| author | Konstantinos Psychogyios Helen C. Leligou Filisia Melissari Stavroula Bourou Zacharias Anastasakis Theodore Zahariadis |
| author_facet | Konstantinos Psychogyios Helen C. Leligou Filisia Melissari Stavroula Bourou Zacharias Anastasakis Theodore Zahariadis |
| author_sort | Konstantinos Psychogyios |
| collection | DOAJ |
| description | Neural Style Transfer (NST) is a popular technique of computer vision where the content of an image is blended with the style of another, which results in a fused image with certain properties of both original images. This approach has practical applications in various domains and has garnered significant attention in both industry and academia. An interesting application of this technique is segmented style transfer where a segmentation algorithm is used to locate objects within an image and then the style transfer method is performed locally, producing images with different styles for different objects. This approach opens up possibilities for creating visually striking compositions by seamlessly blending various artistic styles onto specific objects within an image, allowing for a new level of creative expression. This paper proposes a novel method that combines Segment Anything Model (SAM), a state-of-the-art vision transformer-based image segmentation model developed by Facebook, with style transfer. Our approach includes performing localized style transfer in selected segmentation regions of an image using classical style transfer algorithms. To ensure smooth transitions between the stylized and non-stylized border we also develop our loss function with a border smoothing technique. Experimental results demonstrate the robustness and effectiveness of the proposed methodology, including the ability to infuse multiple artistic styles into different objects within an image. The contributions of this work include integrating SAM with style transfer, proposing a novel loss function, evaluating the segmented style transfer in multiple content regions, comparing with state-of-the-art approaches, and experimenting with multiple style images for diverse stylization. Our primary focus centers on creating a model that serves as a digital painter across a wide range of image genres and artistic styles. |
| first_indexed | 2024-03-11T23:37:20Z |
| format | Article |
| id | doaj.art-d5c6f3cad6864b15ba2ff9d8cc727aad |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T23:37:20Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-d5c6f3cad6864b15ba2ff9d8cc727aad2023-09-19T23:01:20ZengIEEEIEEE Access2169-35362023-01-011110025610026710.1109/ACCESS.2023.331523510250775SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM)Konstantinos Psychogyios0https://orcid.org/0000-0002-9971-9271Helen C. Leligou1https://orcid.org/0000-0002-1489-1495Filisia Melissari2https://orcid.org/0009-0003-9658-9488Stavroula Bourou3https://orcid.org/0000-0001-6122-4161Zacharias Anastasakis4https://orcid.org/0009-0001-9496-0103Theodore Zahariadis5https://orcid.org/0000-0002-2408-4582Synelixis Solutions S.A, Chalkida, GreeceNetcompany-Intrasoft S.A, Paiania, GreeceSynelixis Solutions S.A, Chalkida, GreeceSynelixis Solutions S.A, Chalkida, GreeceSynelixis Solutions S.A, Chalkida, GreeceSynelixis Solutions S.A, Chalkida, GreeceNeural Style Transfer (NST) is a popular technique of computer vision where the content of an image is blended with the style of another, which results in a fused image with certain properties of both original images. This approach has practical applications in various domains and has garnered significant attention in both industry and academia. An interesting application of this technique is segmented style transfer where a segmentation algorithm is used to locate objects within an image and then the style transfer method is performed locally, producing images with different styles for different objects. This approach opens up possibilities for creating visually striking compositions by seamlessly blending various artistic styles onto specific objects within an image, allowing for a new level of creative expression. This paper proposes a novel method that combines Segment Anything Model (SAM), a state-of-the-art vision transformer-based image segmentation model developed by Facebook, with style transfer. Our approach includes performing localized style transfer in selected segmentation regions of an image using classical style transfer algorithms. To ensure smooth transitions between the stylized and non-stylized border we also develop our loss function with a border smoothing technique. Experimental results demonstrate the robustness and effectiveness of the proposed methodology, including the ability to infuse multiple artistic styles into different objects within an image. The contributions of this work include integrating SAM with style transfer, proposing a novel loss function, evaluating the segmented style transfer in multiple content regions, comparing with state-of-the-art approaches, and experimenting with multiple style images for diverse stylization. Our primary focus centers on creating a model that serves as a digital painter across a wide range of image genres and artistic styles.https://ieeexplore.ieee.org/document/10250775/Segment anything modelsegment anythingsegmentationmachine learningstyle transfer |
| spellingShingle | Konstantinos Psychogyios Helen C. Leligou Filisia Melissari Stavroula Bourou Zacharias Anastasakis Theodore Zahariadis SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) IEEE Access Segment anything model segment anything segmentation machine learning style transfer |
| title | SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) |
| title_full | SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) |
| title_fullStr | SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) |
| title_full_unstemmed | SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) |
| title_short | SAMStyler: Enhancing Visual Creativity With Neural Style Transfer and Segment Anything Model (SAM) |
| title_sort | samstyler enhancing visual creativity with neural style transfer and segment anything model sam |
| topic | Segment anything model segment anything segmentation machine learning style transfer |
| url | https://ieeexplore.ieee.org/document/10250775/ |
| work_keys_str_mv | AT konstantinospsychogyios samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam AT helencleligou samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam AT filisiamelissari samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam AT stavroulabourou samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam AT zachariasanastasakis samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam AT theodorezahariadis samstylerenhancingvisualcreativitywithneuralstyletransferandsegmentanythingmodelsam |