Multimodal Emotional Classification Based on Meaningful Learning
Emotion recognition has become one of the most researched subjects in the scientific community, especially in the human–computer interface field. Decades of scientific research have been conducted on unimodal emotion analysis, whereas recent contributions concentrate on multimodal emotion recognitio...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-09-01
|
| Series: | Big Data and Cognitive Computing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-2289/6/3/95 |
| _version_ | 1827663482539999232 |
|---|---|
| author | Hajar Filali Jamal Riffi Chafik Boulealam Mohamed Adnane Mahraz Hamid Tairi |
| author_facet | Hajar Filali Jamal Riffi Chafik Boulealam Mohamed Adnane Mahraz Hamid Tairi |
| author_sort | Hajar Filali |
| collection | DOAJ |
| description | Emotion recognition has become one of the most researched subjects in the scientific community, especially in the human–computer interface field. Decades of scientific research have been conducted on unimodal emotion analysis, whereas recent contributions concentrate on multimodal emotion recognition. These efforts have achieved great success in terms of accuracy in diverse areas of Deep Learning applications. To achieve better performance for multimodal emotion recognition systems, we exploit Meaningful Neural Network Effectiveness to enable emotion prediction during a conversation. Using the text and the audio modalities, we proposed feature extraction methods based on Deep Learning. Then, the bimodal modality that is created following the fusion of the text and audio features is used. The feature vectors from these three modalities are assigned to feed a Meaningful Neural Network to separately learn each characteristic. Its architecture consists of a set of neurons for each component of the input vector before combining them all together in the last layer. Our model was evaluated on a multimodal and multiparty dataset for emotion recognition in conversation MELD. The proposed approach reached an accuracy of 86.69%, which significantly outperforms all current multimodal systems. To sum up, several evaluation techniques applied to our work demonstrate the robustness and superiority of our model over other state-of-the-art MELD models. |
| first_indexed | 2024-03-10T00:43:27Z |
| format | Article |
| id | doaj.art-b62a626ddc6d4daeadca3c5162655441 |
| institution | Directory Open Access Journal |
| issn | 2504-2289 |
| language | English |
| last_indexed | 2024-03-10T00:43:27Z |
| publishDate | 2022-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Big Data and Cognitive Computing |
| spelling | doaj.art-b62a626ddc6d4daeadca3c51626554412023-11-23T15:03:50ZengMDPI AGBig Data and Cognitive Computing2504-22892022-09-01639510.3390/bdcc6030095Multimodal Emotional Classification Based on Meaningful LearningHajar Filali0Jamal Riffi1Chafik Boulealam2Mohamed Adnane Mahraz3Hamid Tairi4Laboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, MoroccoLaboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, MoroccoLaboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, MoroccoLaboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, MoroccoLaboratory of Computer Science, Signals, Automation and Cognitivism (LISAC), Department of Computer Science, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez 30000, MoroccoEmotion recognition has become one of the most researched subjects in the scientific community, especially in the human–computer interface field. Decades of scientific research have been conducted on unimodal emotion analysis, whereas recent contributions concentrate on multimodal emotion recognition. These efforts have achieved great success in terms of accuracy in diverse areas of Deep Learning applications. To achieve better performance for multimodal emotion recognition systems, we exploit Meaningful Neural Network Effectiveness to enable emotion prediction during a conversation. Using the text and the audio modalities, we proposed feature extraction methods based on Deep Learning. Then, the bimodal modality that is created following the fusion of the text and audio features is used. The feature vectors from these three modalities are assigned to feed a Meaningful Neural Network to separately learn each characteristic. Its architecture consists of a set of neurons for each component of the input vector before combining them all together in the last layer. Our model was evaluated on a multimodal and multiparty dataset for emotion recognition in conversation MELD. The proposed approach reached an accuracy of 86.69%, which significantly outperforms all current multimodal systems. To sum up, several evaluation techniques applied to our work demonstrate the robustness and superiority of our model over other state-of-the-art MELD models.https://www.mdpi.com/2504-2289/6/3/95multimodal emotion recognition (MER)deep learning (DL)meaningful neural network (MNN)multimodal and multiparty dataset for emotion recognition in conversations (MELD) |
| spellingShingle | Hajar Filali Jamal Riffi Chafik Boulealam Mohamed Adnane Mahraz Hamid Tairi Multimodal Emotional Classification Based on Meaningful Learning Big Data and Cognitive Computing multimodal emotion recognition (MER) deep learning (DL) meaningful neural network (MNN) multimodal and multiparty dataset for emotion recognition in conversations (MELD) |
| title | Multimodal Emotional Classification Based on Meaningful Learning |
| title_full | Multimodal Emotional Classification Based on Meaningful Learning |
| title_fullStr | Multimodal Emotional Classification Based on Meaningful Learning |
| title_full_unstemmed | Multimodal Emotional Classification Based on Meaningful Learning |
| title_short | Multimodal Emotional Classification Based on Meaningful Learning |
| title_sort | multimodal emotional classification based on meaningful learning |
| topic | multimodal emotion recognition (MER) deep learning (DL) meaningful neural network (MNN) multimodal and multiparty dataset for emotion recognition in conversations (MELD) |
| url | https://www.mdpi.com/2504-2289/6/3/95 |
| work_keys_str_mv | AT hajarfilali multimodalemotionalclassificationbasedonmeaningfullearning AT jamalriffi multimodalemotionalclassificationbasedonmeaningfullearning AT chafikboulealam multimodalemotionalclassificationbasedonmeaningfullearning AT mohamedadnanemahraz multimodalemotionalclassificationbasedonmeaningfullearning AT hamidtairi multimodalemotionalclassificationbasedonmeaningfullearning |