Improving Existing Segmentators Performance with Zero-Shot Segmentators
This paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-10-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/25/11/1502 |
| _version_ | 1827639897339461632 |
|---|---|
| author | Loris Nanni Daniel Fusaro Carlo Fantozzi Alberto Pretto |
| author_facet | Loris Nanni Daniel Fusaro Carlo Fantozzi Alberto Pretto |
| author_sort | Loris Nanni |
| collection | DOAJ |
| description | This paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training. The open-source nature of SAM allows for easy access and implementation. In our experiments, we aim to improve the segmentation performance by providing SAM with checkpoints extracted from the masks produced by mainstream segmentators, and then merging the segmentation masks provided by these two networks. We examine the “oracle” method (as upper bound baseline performance), where segmentation masks are inferred only by SAM with checkpoints extracted from the ground truth. One of the main contributions of this work is the combination (<i>fusion</i>) of the logit segmentation masks produced by the SAM model with the ones provided by specialized segmentation models such as DeepLabv3+ and PVTv2. This combination allows for a consistent improvement in segmentation performance in most of the tested datasets. We exhaustively tested our approach on seven heterogeneous public datasets, obtaining state-of-the-art results in two of them (CAMO and Butterfly) with respect to the current best-performing method with a combination of an ensemble of mainstream segmentator transformers and the SAM segmentator. The results of our study provide valuable insights into the potential of incorporating the SAM segmentator into existing segmentation techniques. We release with this paper the open-source implementation of our method. |
| first_indexed | 2024-03-09T16:50:29Z |
| format | Article |
| id | doaj.art-11e7e435b71649f285db9fa3bf7267ca |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-09T16:50:29Z |
| publishDate | 2023-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-11e7e435b71649f285db9fa3bf7267ca2023-11-24T14:40:56ZengMDPI AGEntropy1099-43002023-10-012511150210.3390/e25111502Improving Existing Segmentators Performance with Zero-Shot SegmentatorsLoris Nanni0Daniel Fusaro1Carlo Fantozzi2Alberto Pretto3Department of Information Engineering, University of Padova, 35122 Padua, ItalyDepartment of Information Engineering, University of Padova, 35122 Padua, ItalyDepartment of Information Engineering, University of Padova, 35122 Padua, ItalyDepartment of Information Engineering, University of Padova, 35122 Padua, ItalyThis paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training. The open-source nature of SAM allows for easy access and implementation. In our experiments, we aim to improve the segmentation performance by providing SAM with checkpoints extracted from the masks produced by mainstream segmentators, and then merging the segmentation masks provided by these two networks. We examine the “oracle” method (as upper bound baseline performance), where segmentation masks are inferred only by SAM with checkpoints extracted from the ground truth. One of the main contributions of this work is the combination (<i>fusion</i>) of the logit segmentation masks produced by the SAM model with the ones provided by specialized segmentation models such as DeepLabv3+ and PVTv2. This combination allows for a consistent improvement in segmentation performance in most of the tested datasets. We exhaustively tested our approach on seven heterogeneous public datasets, obtaining state-of-the-art results in two of them (CAMO and Butterfly) with respect to the current best-performing method with a combination of an ensemble of mainstream segmentator transformers and the SAM segmentator. The results of our study provide valuable insights into the potential of incorporating the SAM segmentator into existing segmentation techniques. We release with this paper the open-source implementation of our method.https://www.mdpi.com/1099-4300/25/11/1502segmentationdeep learningensemblezero-shot segmentator |
| spellingShingle | Loris Nanni Daniel Fusaro Carlo Fantozzi Alberto Pretto Improving Existing Segmentators Performance with Zero-Shot Segmentators Entropy segmentation deep learning ensemble zero-shot segmentator |
| title | Improving Existing Segmentators Performance with Zero-Shot Segmentators |
| title_full | Improving Existing Segmentators Performance with Zero-Shot Segmentators |
| title_fullStr | Improving Existing Segmentators Performance with Zero-Shot Segmentators |
| title_full_unstemmed | Improving Existing Segmentators Performance with Zero-Shot Segmentators |
| title_short | Improving Existing Segmentators Performance with Zero-Shot Segmentators |
| title_sort | improving existing segmentators performance with zero shot segmentators |
| topic | segmentation deep learning ensemble zero-shot segmentator |
| url | https://www.mdpi.com/1099-4300/25/11/1502 |
| work_keys_str_mv | AT lorisnanni improvingexistingsegmentatorsperformancewithzeroshotsegmentators AT danielfusaro improvingexistingsegmentatorsperformancewithzeroshotsegmentators AT carlofantozzi improvingexistingsegmentatorsperformancewithzeroshotsegmentators AT albertopretto improvingexistingsegmentatorsperformancewithzeroshotsegmentators |