Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble
Analyzing the opinions of social media users can lead to a correct understanding of their attitude on different topics. The emotions found in these comments, feedback, or criticisms provide useful indicators for many purposes and can be divided into negative, positive, and neutral categories. Sentim...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | fas |
| Published: |
Semnan University
2023-03-01
|
| Series: | مجله مدل سازی در مهندسی |
| Subjects: | |
| Online Access: | https://modelling.semnan.ac.ir/article_7305_2f7fc5f5239eab37990ba7c46ac995f0.pdf |
| _version_ | 1797296486869893120 |
|---|---|
| author | Alireza Ghorbanali Mohammad Karim Sohrabi Farzin Yaghmaee |
| author_facet | Alireza Ghorbanali Mohammad Karim Sohrabi Farzin Yaghmaee |
| author_sort | Alireza Ghorbanali |
| collection | DOAJ |
| description | Analyzing the opinions of social media users can lead to a correct understanding of their attitude on different topics. The emotions found in these comments, feedback, or criticisms provide useful indicators for many purposes and can be divided into negative, positive, and neutral categories. Sentiment analysis is one of the natural language processing's tasks used in various areas. Some of social media users' opinions is are multimodal and share a combination of multiple media, including text, image and imoji, which provide a useful structure for extracting and better understanding emotions. This paper presents a hybrid transfer learning method using 5 pre-trained models and hybrid convolutional networks for multimodal sentiment analysis. In this method, 2 pre-trained convolutional network-based models are used to extract the properties of images, and 3 other pre-trained models are used to extract the properties of texts and embed words. The extracted features are used in hybrid convolutional networks, and the visual attention mechanism is used to focus on the most important emotional areas of the images and the multi-head attention mechanism is used to highlight the emotional words. The results of the classification of images and texts are combined using the voting technique, and finally the late fusion is used to determine the polarity and the final label. The results of empirical experiments of the proposed model on a standard data set show 96% accuracy. |
| first_indexed | 2024-03-07T22:05:28Z |
| format | Article |
| id | doaj.art-a0c1a6ec6e39493fafad51c149a043bb |
| institution | Directory Open Access Journal |
| issn | 2008-4854 2783-2538 |
| language | fas |
| last_indexed | 2024-03-07T22:05:28Z |
| publishDate | 2023-03-01 |
| publisher | Semnan University |
| record_format | Article |
| series | مجله مدل سازی در مهندسی |
| spelling | doaj.art-a0c1a6ec6e39493fafad51c149a043bb2024-02-23T19:10:30ZfasSemnan Universityمجله مدل سازی در مهندسی2008-48542783-25382023-03-012172839710.22075/jme.2022.25099.21737305Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensembleAlireza Ghorbanali0Mohammad Karim Sohrabi1Farzin Yaghmaee2Semnan branchDepartment of Computer, Semnan branch, Islamic Azad University, Semnan, IranSemnan UniversityAnalyzing the opinions of social media users can lead to a correct understanding of their attitude on different topics. The emotions found in these comments, feedback, or criticisms provide useful indicators for many purposes and can be divided into negative, positive, and neutral categories. Sentiment analysis is one of the natural language processing's tasks used in various areas. Some of social media users' opinions is are multimodal and share a combination of multiple media, including text, image and imoji, which provide a useful structure for extracting and better understanding emotions. This paper presents a hybrid transfer learning method using 5 pre-trained models and hybrid convolutional networks for multimodal sentiment analysis. In this method, 2 pre-trained convolutional network-based models are used to extract the properties of images, and 3 other pre-trained models are used to extract the properties of texts and embed words. The extracted features are used in hybrid convolutional networks, and the visual attention mechanism is used to focus on the most important emotional areas of the images and the multi-head attention mechanism is used to highlight the emotional words. The results of the classification of images and texts are combined using the voting technique, and finally the late fusion is used to determine the polarity and the final label. The results of empirical experiments of the proposed model on a standard data set show 96% accuracy.https://modelling.semnan.ac.ir/article_7305_2f7fc5f5239eab37990ba7c46ac995f0.pdfmultimodal sentiment analysisdeep learningtransfer learningattention mechanismweighted convolutional neural network ensemble |
| spellingShingle | Alireza Ghorbanali Mohammad Karim Sohrabi Farzin Yaghmaee Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble مجله مدل سازی در مهندسی multimodal sentiment analysis deep learning transfer learning attention mechanism weighted convolutional neural network ensemble |
| title | Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| title_full | Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| title_fullStr | Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| title_full_unstemmed | Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| title_short | Multiple transfer learning-based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| title_sort | multiple transfer learning based multimodal sentiment analysis using weighted convolutional neural network ensemble |
| topic | multimodal sentiment analysis deep learning transfer learning attention mechanism weighted convolutional neural network ensemble |
| url | https://modelling.semnan.ac.ir/article_7305_2f7fc5f5239eab37990ba7c46ac995f0.pdf |
| work_keys_str_mv | AT alirezaghorbanali multipletransferlearningbasedmultimodalsentimentanalysisusingweightedconvolutionalneuralnetworkensemble AT mohammadkarimsohrabi multipletransferlearningbasedmultimodalsentimentanalysisusingweightedconvolutionalneuralnetworkensemble AT farzinyaghmaee multipletransferlearningbasedmultimodalsentimentanalysisusingweightedconvolutionalneuralnetworkensemble |