Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism
In recent years, models such as the transformer have demonstrated impressive capabilities in the realm of natural language processing. However, these models are known for their complexity and the substantial training they require. Furthermore, the self-attention mechanism within the transformer, des...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-01-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/12/2/242 |
| _version_ | 1797343041301774336 |
|---|---|
| author | Laith H. Baniata Sangwoo Kang |
| author_facet | Laith H. Baniata Sangwoo Kang |
| author_sort | Laith H. Baniata |
| collection | DOAJ |
| description | In recent years, models such as the transformer have demonstrated impressive capabilities in the realm of natural language processing. However, these models are known for their complexity and the substantial training they require. Furthermore, the self-attention mechanism within the transformer, designed to capture semantic relationships among words in sequences, faces challenges when dealing with short sequences. This limitation hinders its effectiveness in five-polarity Arabic sentiment analysis (SA) tasks. The switch-transformer model has surfaced as a potential substitute. Nevertheless, when employing one-task learning for their training, these models frequently face challenges in presenting exceptional performances and encounter issues when producing resilient latent feature representations, particularly in the context of small-size datasets. This challenge is particularly prominent in the case of the Arabic dialect, which is recognized as a low-resource language. In response to these constraints, this research introduces a novel method for the sentiment analysis of Arabic text. This approach leverages multi-task learning (MTL) in combination with the switch-transformer shared encoder to enhance model adaptability and refine sentence representations. By integrating a mixture of experts (MoE) technique that breaks down the problem into smaller, more manageable sub-problems, the model becomes skilled in managing extended sequences and intricate input–output relationships, thereby benefiting both five-point and three-polarity Arabic sentiment analysis tasks. The proposed model effectively identifies sentiment in Arabic dialect sentences. The empirical results underscore its exceptional performance, with accuracy rates reaching 84.02% for the HARD dataset, 67.89% for the BRAD dataset, and 83.91% for the LABR dataset, as demonstrated by the evaluations conducted on these datasets. |
| first_indexed | 2024-03-08T10:41:56Z |
| format | Article |
| id | doaj.art-0d48dabbdc3e4d729525e31680be2908 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-08T10:41:56Z |
| publishDate | 2024-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-0d48dabbdc3e4d729525e31680be29082024-01-26T17:31:30ZengMDPI AGMathematics2227-73902024-01-0112224210.3390/math12020242Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts MechanismLaith H. Baniata0Sangwoo Kang1School of Computing, Gachon University, Seongnam 13120, Republic of KoreaSchool of Computing, Gachon University, Seongnam 13120, Republic of KoreaIn recent years, models such as the transformer have demonstrated impressive capabilities in the realm of natural language processing. However, these models are known for their complexity and the substantial training they require. Furthermore, the self-attention mechanism within the transformer, designed to capture semantic relationships among words in sequences, faces challenges when dealing with short sequences. This limitation hinders its effectiveness in five-polarity Arabic sentiment analysis (SA) tasks. The switch-transformer model has surfaced as a potential substitute. Nevertheless, when employing one-task learning for their training, these models frequently face challenges in presenting exceptional performances and encounter issues when producing resilient latent feature representations, particularly in the context of small-size datasets. This challenge is particularly prominent in the case of the Arabic dialect, which is recognized as a low-resource language. In response to these constraints, this research introduces a novel method for the sentiment analysis of Arabic text. This approach leverages multi-task learning (MTL) in combination with the switch-transformer shared encoder to enhance model adaptability and refine sentence representations. By integrating a mixture of experts (MoE) technique that breaks down the problem into smaller, more manageable sub-problems, the model becomes skilled in managing extended sequences and intricate input–output relationships, thereby benefiting both five-point and three-polarity Arabic sentiment analysis tasks. The proposed model effectively identifies sentiment in Arabic dialect sentences. The empirical results underscore its exceptional performance, with accuracy rates reaching 84.02% for the HARD dataset, 67.89% for the BRAD dataset, and 83.91% for the LABR dataset, as demonstrated by the evaluations conducted on these datasets.https://www.mdpi.com/2227-7390/12/2/242switch transformermixture of experts (MoE) mechanismsentiment analysis (SA)Arabic dialectsfive-polarityMTL |
| spellingShingle | Laith H. Baniata Sangwoo Kang Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism Mathematics switch transformer mixture of experts (MoE) mechanism sentiment analysis (SA) Arabic dialects five-polarity MTL |
| title | Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism |
| title_full | Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism |
| title_fullStr | Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism |
| title_full_unstemmed | Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism |
| title_short | Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism |
| title_sort | switch transformer sentiment analysis model for arabic dialects that utilizes a mixture of experts mechanism |
| topic | switch transformer mixture of experts (MoE) mechanism sentiment analysis (SA) Arabic dialects five-polarity MTL |
| url | https://www.mdpi.com/2227-7390/12/2/242 |
| work_keys_str_mv | AT laithhbaniata switchtransformersentimentanalysismodelforarabicdialectsthatutilizesamixtureofexpertsmechanism AT sangwookang switchtransformersentimentanalysismodelforarabicdialectsthatutilizesamixtureofexpertsmechanism |