Hateful Memes Detection Based on Multi-Task Learning
With the popularity of posting memes on social platforms, the severe negative impact of hateful memes is growing. As existing detection models have lower detection accuracy than humans, hateful memes detection is still a challenge to statistical learning and artificial intelligence. This paper propo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-11-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/10/23/4525 |
| _version_ | 1797462711976591360 |
|---|---|
| author | Zhiyu Ma Shaowen Yao Liwen Wu Song Gao Yunqi Zhang |
| author_facet | Zhiyu Ma Shaowen Yao Liwen Wu Song Gao Yunqi Zhang |
| author_sort | Zhiyu Ma |
| collection | DOAJ |
| description | With the popularity of posting memes on social platforms, the severe negative impact of hateful memes is growing. As existing detection models have lower detection accuracy than humans, hateful memes detection is still a challenge to statistical learning and artificial intelligence. This paper proposed a multi-task learning method consisting of a primary multimodal task and two unimodal auxiliary tasks to address this issue. We introduced a self-supervised generation strategy in auxiliary tasks to generate unimodal auxiliary labels automatically. Meanwhile, we used BERT and RESNET as the backbone for text and image classification, respectively, and then fusion them with a late fusion method. In the training phase, the backward guidance technique and the adaptive weight adjustment strategy were used to capture the consistency and variability between different modalities, numerically improving the hateful memes detection accuracy and the generalization and robustness of the model. The experiment conducted on the Facebook AI multimodal hateful memes dataset shows that the prediction accuracy of our model outperformed the comparing models. |
| first_indexed | 2024-03-09T17:40:32Z |
| format | Article |
| id | doaj.art-3cff01c784d04d71a93b80272b6fc5a9 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-09T17:40:32Z |
| publishDate | 2022-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-3cff01c784d04d71a93b80272b6fc5a92023-11-24T11:35:01ZengMDPI AGMathematics2227-73902022-11-011023452510.3390/math10234525Hateful Memes Detection Based on Multi-Task LearningZhiyu Ma0Shaowen Yao1Liwen Wu2Song Gao3Yunqi Zhang4Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, ChinaEngineering Research Center of Cyberspace, Yunnan University, Kunming 650091, ChinaEngineering Research Center of Cyberspace, Yunnan University, Kunming 650091, ChinaEngineering Research Center of Cyberspace, Yunnan University, Kunming 650091, ChinaEngineering Research Center of Cyberspace, Yunnan University, Kunming 650091, ChinaWith the popularity of posting memes on social platforms, the severe negative impact of hateful memes is growing. As existing detection models have lower detection accuracy than humans, hateful memes detection is still a challenge to statistical learning and artificial intelligence. This paper proposed a multi-task learning method consisting of a primary multimodal task and two unimodal auxiliary tasks to address this issue. We introduced a self-supervised generation strategy in auxiliary tasks to generate unimodal auxiliary labels automatically. Meanwhile, we used BERT and RESNET as the backbone for text and image classification, respectively, and then fusion them with a late fusion method. In the training phase, the backward guidance technique and the adaptive weight adjustment strategy were used to capture the consistency and variability between different modalities, numerically improving the hateful memes detection accuracy and the generalization and robustness of the model. The experiment conducted on the Facebook AI multimodal hateful memes dataset shows that the prediction accuracy of our model outperformed the comparing models.https://www.mdpi.com/2227-7390/10/23/4525hateful memesdeep learningmultimodal datamulti-task learningself-supervised |
| spellingShingle | Zhiyu Ma Shaowen Yao Liwen Wu Song Gao Yunqi Zhang Hateful Memes Detection Based on Multi-Task Learning Mathematics hateful memes deep learning multimodal data multi-task learning self-supervised |
| title | Hateful Memes Detection Based on Multi-Task Learning |
| title_full | Hateful Memes Detection Based on Multi-Task Learning |
| title_fullStr | Hateful Memes Detection Based on Multi-Task Learning |
| title_full_unstemmed | Hateful Memes Detection Based on Multi-Task Learning |
| title_short | Hateful Memes Detection Based on Multi-Task Learning |
| title_sort | hateful memes detection based on multi task learning |
| topic | hateful memes deep learning multimodal data multi-task learning self-supervised |
| url | https://www.mdpi.com/2227-7390/10/23/4525 |
| work_keys_str_mv | AT zhiyuma hatefulmemesdetectionbasedonmultitasklearning AT shaowenyao hatefulmemesdetectionbasedonmultitasklearning AT liwenwu hatefulmemesdetectionbasedonmultitasklearning AT songgao hatefulmemesdetectionbasedonmultitasklearning AT yunqizhang hatefulmemesdetectionbasedonmultitasklearning |