Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion
With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emoti...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-07-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/15/5528 |
| _version_ | 1797412343020257280 |
|---|---|
| author | Jian Zhao Wenhua Dong Lijuan Shi Wenqian Qiang Zhejun Kuang Dawei Xu Tianbo An |
| author_facet | Jian Zhao Wenhua Dong Lijuan Shi Wenqian Qiang Zhejun Kuang Dawei Xu Tianbo An |
| author_sort | Jian Zhao |
| collection | DOAJ |
| description | With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis. |
| first_indexed | 2024-03-09T05:00:42Z |
| format | Article |
| id | doaj.art-0d3cbabd3b2d4efb99c865fb5e3b043c |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T05:00:42Z |
| publishDate | 2022-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-0d3cbabd3b2d4efb99c865fb5e3b043c2023-12-03T12:59:59ZengMDPI AGSensors1424-82202022-07-012215552810.3390/s22155528Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public OpinionJian Zhao0Wenhua Dong1Lijuan Shi2Wenqian Qiang3Zhejun Kuang4Dawei Xu5Tianbo An6School of Cyber Security, Changchun University, Changchun 130022, ChinaSchool of Cyber Security, Changchun University, Changchun 130022, ChinaJilin Provincial Key Laboratory of Human Health Status Identification and Function Enhancement, Changchun 130022, ChinaSchool of Cyber Security, Changchun University, Changchun 130022, ChinaSchool of Computer Science and Technology, Changchun University, Changchun 130022, ChinaSchool of Cyber Security, Changchun University, Changchun 130022, ChinaSchool of Cyber Security, Changchun University, Changchun 130022, ChinaWith the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis.https://www.mdpi.com/1424-8220/22/15/5528multi-modalityfine-grainedmodel fusionemotion featuresfeature generation |
| spellingShingle | Jian Zhao Wenhua Dong Lijuan Shi Wenqian Qiang Zhejun Kuang Dawei Xu Tianbo An Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion Sensors multi-modality fine-grained model fusion emotion features feature generation |
| title | Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion |
| title_full | Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion |
| title_fullStr | Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion |
| title_full_unstemmed | Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion |
| title_short | Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion |
| title_sort | multimodal feature fusion method for unbalanced sample data in social network public opinion |
| topic | multi-modality fine-grained model fusion emotion features feature generation |
| url | https://www.mdpi.com/1424-8220/22/15/5528 |
| work_keys_str_mv | AT jianzhao multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT wenhuadong multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT lijuanshi multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT wenqianqiang multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT zhejunkuang multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT daweixu multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion AT tianboan multimodalfeaturefusionmethodforunbalancedsampledatainsocialnetworkpublicopinion |