ResMem-Net: memory based deep CNN for image memorability estimation
Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learni...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2021-11-01
|
| Series: | PeerJ Computer Science |
| Subjects: | |
| Online Access: | https://peerj.com/articles/cs-767.pdf |
| _version_ | 1831585748120240128 |
|---|---|
| author | Arockia Praveen Abdulfattah Noorwali Duraimurugan Samiayya Mohammad Zubair Khan Durai Raj Vincent P M Ali Kashif Bashir Vinoth Alagupandi |
| author_facet | Arockia Praveen Abdulfattah Noorwali Duraimurugan Samiayya Mohammad Zubair Khan Durai Raj Vincent P M Ali Kashif Bashir Vinoth Alagupandi |
| author_sort | Arockia Praveen |
| collection | DOAJ |
| description | Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are important for predicting the output because they contain information about the intrinsic properties of the image. The proposed architecture automatically learns visual emotions and saliency, shown by the heatmaps generated using the GradRAM technique. We have also used the heatmaps and results to analyze and answer one of the most important questions in image memorability: “What makes an image memorable?”. The model is trained and evaluated using the publicly available Large-scale Image Memorability dataset (LaMem) from MIT. The results show that the model achieves a rank correlation of 0.679 and a mean squared error of 0.011, which is better than the current state-of-the-art models and is close to human consistency (p = 0.68). The proposed architecture also has a significantly low number of parameters compared to the state-of-the-art architecture, making it memory efficient and suitable for production. |
| first_indexed | 2024-12-17T21:24:07Z |
| format | Article |
| id | doaj.art-de7ec13f51074916a588ceccf0033b8e |
| institution | Directory Open Access Journal |
| issn | 2376-5992 |
| language | English |
| last_indexed | 2024-12-17T21:24:07Z |
| publishDate | 2021-11-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ Computer Science |
| spelling | doaj.art-de7ec13f51074916a588ceccf0033b8e2022-12-21T21:32:05ZengPeerJ Inc.PeerJ Computer Science2376-59922021-11-017e76710.7717/peerj-cs.767ResMem-Net: memory based deep CNN for image memorability estimationArockia Praveen0Abdulfattah Noorwali1Duraimurugan Samiayya2Mohammad Zubair Khan3Durai Raj Vincent P M4Ali Kashif Bashir5Vinoth Alagupandi6Phosphene AI, Madurai, IndiaUmm Al-Qura University, Makkah, Saudi ArabiaOptisol Business Solutions, Chennai, IndiaDepartment of Computer Science, Taibah University, Medina, Saudi ArabiaSchool of Information Technology and Engineering, Vellore Institute of Technology, Vellore, Tamilnadu, IndiaThe Manchester Metropolitan University, Manchester, United KingdomOptisol Business Solutions, Chennai, IndiaImage memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are important for predicting the output because they contain information about the intrinsic properties of the image. The proposed architecture automatically learns visual emotions and saliency, shown by the heatmaps generated using the GradRAM technique. We have also used the heatmaps and results to analyze and answer one of the most important questions in image memorability: “What makes an image memorable?”. The model is trained and evaluated using the publicly available Large-scale Image Memorability dataset (LaMem) from MIT. The results show that the model achieves a rank correlation of 0.679 and a mean squared error of 0.011, which is better than the current state-of-the-art models and is close to human consistency (p = 0.68). The proposed architecture also has a significantly low number of parameters compared to the state-of-the-art architecture, making it memory efficient and suitable for production.https://peerj.com/articles/cs-767.pdfDeep LearningImage MemorabilityVisual EmotionsSaliencyObject Interestingness |
| spellingShingle | Arockia Praveen Abdulfattah Noorwali Duraimurugan Samiayya Mohammad Zubair Khan Durai Raj Vincent P M Ali Kashif Bashir Vinoth Alagupandi ResMem-Net: memory based deep CNN for image memorability estimation PeerJ Computer Science Deep Learning Image Memorability Visual Emotions Saliency Object Interestingness |
| title | ResMem-Net: memory based deep CNN for image memorability estimation |
| title_full | ResMem-Net: memory based deep CNN for image memorability estimation |
| title_fullStr | ResMem-Net: memory based deep CNN for image memorability estimation |
| title_full_unstemmed | ResMem-Net: memory based deep CNN for image memorability estimation |
| title_short | ResMem-Net: memory based deep CNN for image memorability estimation |
| title_sort | resmem net memory based deep cnn for image memorability estimation |
| topic | Deep Learning Image Memorability Visual Emotions Saliency Object Interestingness |
| url | https://peerj.com/articles/cs-767.pdf |
| work_keys_str_mv | AT arockiapraveen resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT abdulfattahnoorwali resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT duraimurugansamiayya resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT mohammadzubairkhan resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT durairajvincentpm resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT alikashifbashir resmemnetmemorybaseddeepcnnforimagememorabilityestimation AT vinothalagupandi resmemnetmemorybaseddeepcnnforimagememorabilityestimation |