Social Image Captioning: Exploring Visual Attention and User Attention
Image captioning with a natural language has been an emerging trend. However, the social image, associated with a set of user-contributed tags, has been rarely investigated for a similar task. The user-contributed tags, which could reflect the user attention, have been neglected in conventional imag...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2018-02-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | http://www.mdpi.com/1424-8220/18/2/646 |
| _version_ | 1798039025694539776 |
|---|---|
| author | Leiquan Wang Xiaoliang Chu Weishan Zhang Yiwei Wei Weichen Sun Chunlei Wu |
| author_facet | Leiquan Wang Xiaoliang Chu Weishan Zhang Yiwei Wei Weichen Sun Chunlei Wu |
| author_sort | Leiquan Wang |
| collection | DOAJ |
| description | Image captioning with a natural language has been an emerging trend. However, the social image, associated with a set of user-contributed tags, has been rarely investigated for a similar task. The user-contributed tags, which could reflect the user attention, have been neglected in conventional image captioning. Most existing image captioning models cannot be applied directly to social image captioning. In this work, a dual attention model is proposed for social image captioning by combining the visual attention and user attention simultaneously.Visual attention is used to compress a large mount of salient visual information, while user attention is applied to adjust the description of the social images with user-contributed tags. Experiments conducted on the Microsoft (MS) COCO dataset demonstrate the superiority of the proposed method of dual attention. |
| first_indexed | 2024-04-11T21:48:15Z |
| format | Article |
| id | doaj.art-06ce2509ae7e48b5b5f40e54686795de |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-04-11T21:48:15Z |
| publishDate | 2018-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-06ce2509ae7e48b5b5f40e54686795de2022-12-22T04:01:19ZengMDPI AGSensors1424-82202018-02-0118264610.3390/s18020646s18020646Social Image Captioning: Exploring Visual Attention and User AttentionLeiquan Wang0Xiaoliang Chu1Weishan Zhang2Yiwei Wei3Weichen Sun4Chunlei Wu5College of Computer & Communication Engineering, China University of Petroleum (East China), Qingdao 266555, ChinaCollege of Computer & Communication Engineering, China University of Petroleum (East China), Qingdao 266555, ChinaCollege of Computer & Communication Engineering, China University of Petroleum (East China), Qingdao 266555, ChinaCollege of Computer & Communication Engineering, China University of Petroleum (East China), Qingdao 266555, ChinaFirst Research Institute of the Ministry of Public Security of PRC, Beijing 100048, ChinaCollege of Computer & Communication Engineering, China University of Petroleum (East China), Qingdao 266555, ChinaImage captioning with a natural language has been an emerging trend. However, the social image, associated with a set of user-contributed tags, has been rarely investigated for a similar task. The user-contributed tags, which could reflect the user attention, have been neglected in conventional image captioning. Most existing image captioning models cannot be applied directly to social image captioning. In this work, a dual attention model is proposed for social image captioning by combining the visual attention and user attention simultaneously.Visual attention is used to compress a large mount of salient visual information, while user attention is applied to adjust the description of the social images with user-contributed tags. Experiments conducted on the Microsoft (MS) COCO dataset demonstrate the superiority of the proposed method of dual attention.http://www.mdpi.com/1424-8220/18/2/646social image captioninguser-contributed tagsuser attentionvisual attention |
| spellingShingle | Leiquan Wang Xiaoliang Chu Weishan Zhang Yiwei Wei Weichen Sun Chunlei Wu Social Image Captioning: Exploring Visual Attention and User Attention Sensors social image captioning user-contributed tags user attention visual attention |
| title | Social Image Captioning: Exploring Visual Attention and User Attention |
| title_full | Social Image Captioning: Exploring Visual Attention and User Attention |
| title_fullStr | Social Image Captioning: Exploring Visual Attention and User Attention |
| title_full_unstemmed | Social Image Captioning: Exploring Visual Attention and User Attention |
| title_short | Social Image Captioning: Exploring Visual Attention and User Attention |
| title_sort | social image captioning exploring visual attention and user attention |
| topic | social image captioning user-contributed tags user attention visual attention |
| url | http://www.mdpi.com/1424-8220/18/2/646 |
| work_keys_str_mv | AT leiquanwang socialimagecaptioningexploringvisualattentionanduserattention AT xiaoliangchu socialimagecaptioningexploringvisualattentionanduserattention AT weishanzhang socialimagecaptioningexploringvisualattentionanduserattention AT yiweiwei socialimagecaptioningexploringvisualattentionanduserattention AT weichensun socialimagecaptioningexploringvisualattentionanduserattention AT chunleiwu socialimagecaptioningexploringvisualattentionanduserattention |