Contour detection network for zero-shot sketch-based image retrieval
Abstract Zero-shot sketch-based image retrieval (ZS-SBIR) is a challenging task that involves searching natural images related to a given hand-drawn sketch under the zero-shot scene. The previous approach projected image and sketch features into a low-dimensional common space for retrieval, and used...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2023-06-01
|
| Series: | Complex & Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s40747-023-01096-2 |
| _version_ | 1797647124953825280 |
|---|---|
| author | Qing Zhang Jing Zhang Xiangdong Su Feilong Bao Guanglai Gao |
| author_facet | Qing Zhang Jing Zhang Xiangdong Su Feilong Bao Guanglai Gao |
| author_sort | Qing Zhang |
| collection | DOAJ |
| description | Abstract Zero-shot sketch-based image retrieval (ZS-SBIR) is a challenging task that involves searching natural images related to a given hand-drawn sketch under the zero-shot scene. The previous approach projected image and sketch features into a low-dimensional common space for retrieval, and used semantic features to transfer the knowledge of seen to unseen classes. However, it is not effective enough to align multimodal features when projecting them into a common space, since the styles and contents of sketches and natural images are different and they are not one-to-one correspondence. To solve this problem, we propose a novel three-branch joint training network with contour detection network (called CDNNet) for the ZS-SBIR task, which uses contour maps as a bridge to align sketches and natural images to alleviate the domain gap. Specifically, we use semantic metrics to constrain the relationship between contour images and natural images and between contour images and sketches, so that natural image and sketch features can be aligned in the common space. Meanwhile, we further employ second-order attention to capture target subject information to increase the performance of retrieval descriptors. In addition, we use a teacher model and word embedding method to transfer the knowledge of the seen to the unseen classes. Extensive experiments on two large-scale datasets demonstrate that our proposed approach outperforms state-of-the-art CNN-based models: it improves by 2.6% on the Sketchy and 1.2% on TU-Berlin datasets in terms of mAP. |
| first_indexed | 2024-03-11T15:12:48Z |
| format | Article |
| id | doaj.art-77981a57900047f8911038fb6aed68e4 |
| institution | Directory Open Access Journal |
| issn | 2199-4536 2198-6053 |
| language | English |
| last_indexed | 2024-03-11T15:12:48Z |
| publishDate | 2023-06-01 |
| publisher | Springer |
| record_format | Article |
| series | Complex & Intelligent Systems |
| spelling | doaj.art-77981a57900047f8911038fb6aed68e42023-10-29T12:41:03ZengSpringerComplex & Intelligent Systems2199-45362198-60532023-06-01966781679510.1007/s40747-023-01096-2Contour detection network for zero-shot sketch-based image retrievalQing Zhang0Jing Zhang1Xiangdong Su2Feilong Bao3Guanglai Gao4College of Computer Science, Inner Mongolia UniversityCollege of Computer Science, Inner Mongolia UniversityCollege of Computer Science, Inner Mongolia UniversityCollege of Computer Science, Inner Mongolia UniversityCollege of Computer Science, Inner Mongolia UniversityAbstract Zero-shot sketch-based image retrieval (ZS-SBIR) is a challenging task that involves searching natural images related to a given hand-drawn sketch under the zero-shot scene. The previous approach projected image and sketch features into a low-dimensional common space for retrieval, and used semantic features to transfer the knowledge of seen to unseen classes. However, it is not effective enough to align multimodal features when projecting them into a common space, since the styles and contents of sketches and natural images are different and they are not one-to-one correspondence. To solve this problem, we propose a novel three-branch joint training network with contour detection network (called CDNNet) for the ZS-SBIR task, which uses contour maps as a bridge to align sketches and natural images to alleviate the domain gap. Specifically, we use semantic metrics to constrain the relationship between contour images and natural images and between contour images and sketches, so that natural image and sketch features can be aligned in the common space. Meanwhile, we further employ second-order attention to capture target subject information to increase the performance of retrieval descriptors. In addition, we use a teacher model and word embedding method to transfer the knowledge of the seen to the unseen classes. Extensive experiments on two large-scale datasets demonstrate that our proposed approach outperforms state-of-the-art CNN-based models: it improves by 2.6% on the Sketchy and 1.2% on TU-Berlin datasets in terms of mAP.https://doi.org/10.1007/s40747-023-01096-2Sketch-based image retrievalZero-shot learningCross-modal retrievalContour detection |
| spellingShingle | Qing Zhang Jing Zhang Xiangdong Su Feilong Bao Guanglai Gao Contour detection network for zero-shot sketch-based image retrieval Complex & Intelligent Systems Sketch-based image retrieval Zero-shot learning Cross-modal retrieval Contour detection |
| title | Contour detection network for zero-shot sketch-based image retrieval |
| title_full | Contour detection network for zero-shot sketch-based image retrieval |
| title_fullStr | Contour detection network for zero-shot sketch-based image retrieval |
| title_full_unstemmed | Contour detection network for zero-shot sketch-based image retrieval |
| title_short | Contour detection network for zero-shot sketch-based image retrieval |
| title_sort | contour detection network for zero shot sketch based image retrieval |
| topic | Sketch-based image retrieval Zero-shot learning Cross-modal retrieval Contour detection |
| url | https://doi.org/10.1007/s40747-023-01096-2 |
| work_keys_str_mv | AT qingzhang contourdetectionnetworkforzeroshotsketchbasedimageretrieval AT jingzhang contourdetectionnetworkforzeroshotsketchbasedimageretrieval AT xiangdongsu contourdetectionnetworkforzeroshotsketchbasedimageretrieval AT feilongbao contourdetectionnetworkforzeroshotsketchbasedimageretrieval AT guanglaigao contourdetectionnetworkforzeroshotsketchbasedimageretrieval |