Zero-Shot Semantic Segmentation Using Relation Network
Zero-shot learning (ZSL) is widely studied in recent years to solve the problem of lacking annotations. Currently, most studies on ZSL are for image classification and object detection. But, zero-shot semantic segmentation, pixel level classification, is still at its early stage. Therefore, this wor...
Main Authors: | , |
---|---|
Format: | Article |
Language: | English |
Published: |
FRUCT
2021-01-01
|
Series: | Proceedings of the XXth Conference of Open Innovations Association FRUCT |
Subjects: | |
Online Access: | https://www.fruct.org/publications/fruct28/files/Zha.pdf |
Summary: | Zero-shot learning (ZSL) is widely studied in recent years to solve the problem of lacking annotations. Currently, most studies on ZSL are for image classification and object detection. But, zero-shot semantic segmentation, pixel level classification, is still at its early stage. Therefore, this work proposes to extend a zero-shot image classification model, Relation Network (RN), to semantic segmentation tasks. We modified the structure of RN based on other state-of-the-arts semantic segmentation models (i.e. U-Net and DeepLab) and utilizes word embeddings from Caltech-UCSD Birds 200-2011 attributes and natural language processing models (i.e. word2vec and fastText). Because meta-learning is limited to binary tasks, this work proposes to join multiple binary semantic segmentation pipelines for multi-class semantic segmentation. It is proved by experiments that RN could improve accuracy of U-Net with the help of semantic side information on binary semantic segmentation and it could also be applied on multi-class semantic segmentation with simpler structure than the baseline model, SPNet, but higher accuracy under ZSL setting. However, the capability of RN under generalized zero-shot learning (GZSL) setting still needs improvement. We also studied on how different word embeddings, network structures and data affect RN and what could be done to improve its results. |
---|---|
ISSN: | 2305-7254 2343-0737 |