An image caption model based on attention mechanism and deep reinforcement learning
Image caption technology aims to convert visual features of images, extracted by computers, into meaningful semantic information. Therefore, the computers can generate text descriptions that resemble human perception, enabling tasks such as image classification, retrieval, and analysis. In recent ye...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-10-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2023.1270850/full |
| _version_ | 1797665931077353472 |
|---|---|
| author | Tong Bai Sen Zhou Yu Pang Jiasai Luo Huiqian Wang Ya Du |
| author_facet | Tong Bai Sen Zhou Yu Pang Jiasai Luo Huiqian Wang Ya Du |
| author_sort | Tong Bai |
| collection | DOAJ |
| description | Image caption technology aims to convert visual features of images, extracted by computers, into meaningful semantic information. Therefore, the computers can generate text descriptions that resemble human perception, enabling tasks such as image classification, retrieval, and analysis. In recent years, the performance of image caption has been significantly enhanced with the introduction of encoder-decoder architecture in machine translation and the utilization of deep neural networks. However, several challenges still persist in this domain. Therefore, this paper proposes a novel method to address the issue of visual information loss and non-dynamic adjustment of input images during decoding. We introduce a guided decoding network that establishes a connection between the encoding and decoding parts. Through this connection, encoding information can provide guidance to the decoding process, facilitating automatic adjustment of the decoding information. In addition, Dense Convolutional Network (DenseNet) and Multiple Instance Learning (MIL) are adopted in the image encoder, and Nested Long Short-Term Memory (NLSTM) is utilized as the decoder to enhance the extraction and parsing capability of image information during the encoding and decoding process. In order to further improve the performance of our image caption model, this study incorporates an attention mechanism to focus details and constructs a double-layer decoding structure, which facilitates the enhancement of the model in terms of providing more detailed descriptions and enriched semantic information. Furthermore, the Deep Reinforcement Learning (DRL) method is employed to train the model by directly optimizing the identical set of evaluation indexes, which solves the problem of inconsistent training and evaluation standards. Finally, the model is trained and tested on MS COCO and Flickr 30 k datasets, and the results show that the model has improved compared with commonly used models in the evaluation indicators such as BLEU, METEOR and CIDEr. |
| first_indexed | 2024-03-11T19:51:13Z |
| format | Article |
| id | doaj.art-3da35ecc04c547aa8d75c835a3eb5293 |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-03-11T19:51:13Z |
| publishDate | 2023-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-3da35ecc04c547aa8d75c835a3eb52932023-10-05T06:44:55ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2023-10-011710.3389/fnins.2023.12708501270850An image caption model based on attention mechanism and deep reinforcement learningTong Bai0Sen Zhou1Yu Pang2Jiasai Luo3Huiqian Wang4Ya Du5School of Optoelectronic Engineering, Chongqing University of Posts and Telecommunications, Chongqing, ChinaChongqing Academy of Metrology and Quality Inspection, Chongqing, ChinaSchool of Optoelectronic Engineering, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSchool of Optoelectronic Engineering, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSchool of Optoelectronic Engineering, Chongqing University of Posts and Telecommunications, Chongqing, ChinaDepartment of Peripheral Vascular (Wound Repair), Chongqing Hospital of Traditional Chinese Medicine, Chongqing, ChinaImage caption technology aims to convert visual features of images, extracted by computers, into meaningful semantic information. Therefore, the computers can generate text descriptions that resemble human perception, enabling tasks such as image classification, retrieval, and analysis. In recent years, the performance of image caption has been significantly enhanced with the introduction of encoder-decoder architecture in machine translation and the utilization of deep neural networks. However, several challenges still persist in this domain. Therefore, this paper proposes a novel method to address the issue of visual information loss and non-dynamic adjustment of input images during decoding. We introduce a guided decoding network that establishes a connection between the encoding and decoding parts. Through this connection, encoding information can provide guidance to the decoding process, facilitating automatic adjustment of the decoding information. In addition, Dense Convolutional Network (DenseNet) and Multiple Instance Learning (MIL) are adopted in the image encoder, and Nested Long Short-Term Memory (NLSTM) is utilized as the decoder to enhance the extraction and parsing capability of image information during the encoding and decoding process. In order to further improve the performance of our image caption model, this study incorporates an attention mechanism to focus details and constructs a double-layer decoding structure, which facilitates the enhancement of the model in terms of providing more detailed descriptions and enriched semantic information. Furthermore, the Deep Reinforcement Learning (DRL) method is employed to train the model by directly optimizing the identical set of evaluation indexes, which solves the problem of inconsistent training and evaluation standards. Finally, the model is trained and tested on MS COCO and Flickr 30 k datasets, and the results show that the model has improved compared with commonly used models in the evaluation indicators such as BLEU, METEOR and CIDEr.https://www.frontiersin.org/articles/10.3389/fnins.2023.1270850/fullimage captionencoder-decoder architecturedeep neural networksattention mechanismdeep reinforcement learning |
| spellingShingle | Tong Bai Sen Zhou Yu Pang Jiasai Luo Huiqian Wang Ya Du An image caption model based on attention mechanism and deep reinforcement learning Frontiers in Neuroscience image caption encoder-decoder architecture deep neural networks attention mechanism deep reinforcement learning |
| title | An image caption model based on attention mechanism and deep reinforcement learning |
| title_full | An image caption model based on attention mechanism and deep reinforcement learning |
| title_fullStr | An image caption model based on attention mechanism and deep reinforcement learning |
| title_full_unstemmed | An image caption model based on attention mechanism and deep reinforcement learning |
| title_short | An image caption model based on attention mechanism and deep reinforcement learning |
| title_sort | image caption model based on attention mechanism and deep reinforcement learning |
| topic | image caption encoder-decoder architecture deep neural networks attention mechanism deep reinforcement learning |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2023.1270850/full |
| work_keys_str_mv | AT tongbai animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT senzhou animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT yupang animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT jiasailuo animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT huiqianwang animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT yadu animagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT tongbai imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT senzhou imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT yupang imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT jiasailuo imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT huiqianwang imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning AT yadu imagecaptionmodelbasedonattentionmechanismanddeepreinforcementlearning |