Self-supervised utterance order prediction for emotion recognition in conversations
As the order of the utterances in a conversation changes, the meaning of the utterance also changes, and sometimes, this will cause different semantics or emotions. However, the existing representation learning models do not pay close attention to capturing the internal semantic differences of utter...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/175849 |
| _version_ | 1811690378207690752 |
|---|---|
| author | Jiang, Dazhi Liu, Hao Tu, Geng Wei, Runguo Cambria, Erik |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Jiang, Dazhi Liu, Hao Tu, Geng Wei, Runguo Cambria, Erik |
| author_sort | Jiang, Dazhi |
| collection | NTU |
| description | As the order of the utterances in a conversation changes, the meaning of the utterance also changes, and sometimes, this will cause different semantics or emotions. However, the existing representation learning models do not pay close attention to capturing the internal semantic differences of utterance caused by the change of utterance order. Based on this, we build a self-supervised utterance order prediction approach to learn the logical order of utterance, which helps understand the deep semantic relationship between adjacent utterances. Specially, the utterance binary composed of two adjacent utterances, which are ordered or disordered, is fed to the self-supervised model so that the self-supervised model can obtain firm representation learning ability for the semantic differences of the adjacent sentences. The self-supervised method is applied to the downstream conversation emotion recognition task to test the value of the approach. The features extracted from the self-supervised model are fused with the multimodal features to obtain a richer utterance representation. After that, emotion recognition models are applied to two different datasets. The experiment results show that our proposed approach outperforms the current state of the art on ERC benchmark datasets. |
| first_indexed | 2024-10-01T06:03:02Z |
| format | Journal Article |
| id | ntu-10356/175849 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T06:03:02Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1758492024-05-08T02:32:18Z Self-supervised utterance order prediction for emotion recognition in conversations Jiang, Dazhi Liu, Hao Tu, Geng Wei, Runguo Cambria, Erik School of Computer Science and Engineering Computer and Information Science Self-supervised learning Utterance order prediction As the order of the utterances in a conversation changes, the meaning of the utterance also changes, and sometimes, this will cause different semantics or emotions. However, the existing representation learning models do not pay close attention to capturing the internal semantic differences of utterance caused by the change of utterance order. Based on this, we build a self-supervised utterance order prediction approach to learn the logical order of utterance, which helps understand the deep semantic relationship between adjacent utterances. Specially, the utterance binary composed of two adjacent utterances, which are ordered or disordered, is fed to the self-supervised model so that the self-supervised model can obtain firm representation learning ability for the semantic differences of the adjacent sentences. The self-supervised method is applied to the downstream conversation emotion recognition task to test the value of the approach. The features extracted from the self-supervised model are fused with the multimodal features to obtain a richer utterance representation. After that, emotion recognition models are applied to two different datasets. The experiment results show that our proposed approach outperforms the current state of the art on ERC benchmark datasets. This research is funded by the National Natural Science Foundation of China (62372283, 62206163), Science and Technology Major Project of Guangdong Province, China (STKJ2021005, STKJ202209002, STKJ2023076), Natural Science Foundation of Guangdong Province, China (2019A1515010943). 2024-05-08T02:32:18Z 2024-05-08T02:32:18Z 2024 Journal Article Jiang, D., Liu, H., Tu, G., Wei, R. & Cambria, E. (2024). Self-supervised utterance order prediction for emotion recognition in conversations. Neurocomputing, 577, 127370-. https://dx.doi.org/10.1016/j.neucom.2024.127370 0925-2312 https://hdl.handle.net/10356/175849 10.1016/j.neucom.2024.127370 2-s2.0-85185453149 577 127370 en Neurocomputing © 2024 Elsevier B.V. All rights reserved. |
| spellingShingle | Computer and Information Science Self-supervised learning Utterance order prediction Jiang, Dazhi Liu, Hao Tu, Geng Wei, Runguo Cambria, Erik Self-supervised utterance order prediction for emotion recognition in conversations |
| title | Self-supervised utterance order prediction for emotion recognition in conversations |
| title_full | Self-supervised utterance order prediction for emotion recognition in conversations |
| title_fullStr | Self-supervised utterance order prediction for emotion recognition in conversations |
| title_full_unstemmed | Self-supervised utterance order prediction for emotion recognition in conversations |
| title_short | Self-supervised utterance order prediction for emotion recognition in conversations |
| title_sort | self supervised utterance order prediction for emotion recognition in conversations |
| topic | Computer and Information Science Self-supervised learning Utterance order prediction |
| url | https://hdl.handle.net/10356/175849 |
| work_keys_str_mv | AT jiangdazhi selfsupervisedutteranceorderpredictionforemotionrecognitioninconversations AT liuhao selfsupervisedutteranceorderpredictionforemotionrecognitioninconversations AT tugeng selfsupervisedutteranceorderpredictionforemotionrecognitioninconversations AT weirunguo selfsupervisedutteranceorderpredictionforemotionrecognitioninconversations AT cambriaerik selfsupervisedutteranceorderpredictionforemotionrecognitioninconversations |