Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5
Due to the promising performance of pre-trained language models for task-oriented dialogue systems (DS) in English, some efforts to provide multilingual models for task-oriented DS in low-resource languages have emerged. These efforts still face a long-standing challenge due to the lack of high-qual...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-02-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/10/5/746 |
| _version_ | 1797474522305134592 |
|---|---|
| author | Ahlam Fuad Maha Al-Yahya |
| author_facet | Ahlam Fuad Maha Al-Yahya |
| author_sort | Ahlam Fuad |
| collection | DOAJ |
| description | Due to the promising performance of pre-trained language models for task-oriented dialogue systems (DS) in English, some efforts to provide multilingual models for task-oriented DS in low-resource languages have emerged. These efforts still face a long-standing challenge due to the lack of high-quality data for these languages, especially Arabic. To circumvent the cost and time-intensive data collection and annotation, cross-lingual transfer learning can be used when few training data are available in the low-resource target language. Therefore, this study aims to explore the effectiveness of cross-lingual transfer learning in building an end-to-end Arabic task-oriented DS using the mT5 transformer model. We use the Arabic task-oriented dialogue dataset (Arabic-TOD) in the training and testing of the model. We present the cross-lingual transfer learning deployed with three different approaches: mSeq2Seq, Cross-lingual Pre-training (CPT), and Mixed-Language Pre-training (MLT). We obtain good results for our model compared to the literature for Chinese language using the same settings. Furthermore, cross-lingual transfer learning deployed with the MLT approach outperform the other two approaches. Finally, we show that our results can be improved by increasing the training dataset size. |
| first_indexed | 2024-03-09T20:31:18Z |
| format | Article |
| id | doaj.art-a3bbfc1c3fc946938b2b8418aa682a06 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-09T20:31:18Z |
| publishDate | 2022-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-a3bbfc1c3fc946938b2b8418aa682a062023-11-23T23:22:59ZengMDPI AGMathematics2227-73902022-02-0110574610.3390/math10050746Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5Ahlam Fuad0Maha Al-Yahya1Department of Information Technology, College of Computer and Information Sciences, King Saud University, P.O. Box 145111, Riyadh 4545, Saudi ArabiaDepartment of Information Technology, College of Computer and Information Sciences, King Saud University, P.O. Box 145111, Riyadh 4545, Saudi ArabiaDue to the promising performance of pre-trained language models for task-oriented dialogue systems (DS) in English, some efforts to provide multilingual models for task-oriented DS in low-resource languages have emerged. These efforts still face a long-standing challenge due to the lack of high-quality data for these languages, especially Arabic. To circumvent the cost and time-intensive data collection and annotation, cross-lingual transfer learning can be used when few training data are available in the low-resource target language. Therefore, this study aims to explore the effectiveness of cross-lingual transfer learning in building an end-to-end Arabic task-oriented DS using the mT5 transformer model. We use the Arabic task-oriented dialogue dataset (Arabic-TOD) in the training and testing of the model. We present the cross-lingual transfer learning deployed with three different approaches: mSeq2Seq, Cross-lingual Pre-training (CPT), and Mixed-Language Pre-training (MLT). We obtain good results for our model compared to the literature for Chinese language using the same settings. Furthermore, cross-lingual transfer learning deployed with the MLT approach outperform the other two approaches. Finally, we show that our results can be improved by increasing the training dataset size.https://www.mdpi.com/2227-7390/10/5/746cross-lingual transfer learningtask-oriented dialogue systemsArabic languagemixed-language pre-trainingmultilingual transformer modelmT5 |
| spellingShingle | Ahlam Fuad Maha Al-Yahya Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 Mathematics cross-lingual transfer learning task-oriented dialogue systems Arabic language mixed-language pre-training multilingual transformer model mT5 |
| title | Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 |
| title_full | Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 |
| title_fullStr | Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 |
| title_full_unstemmed | Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 |
| title_short | Cross-Lingual Transfer Learning for Arabic Task-Oriented Dialogue Systems Using Multilingual Transformer Model mT5 |
| title_sort | cross lingual transfer learning for arabic task oriented dialogue systems using multilingual transformer model mt5 |
| topic | cross-lingual transfer learning task-oriented dialogue systems Arabic language mixed-language pre-training multilingual transformer model mT5 |
| url | https://www.mdpi.com/2227-7390/10/5/746 |
| work_keys_str_mv | AT ahlamfuad crosslingualtransferlearningforarabictaskorienteddialoguesystemsusingmultilingualtransformermodelmt5 AT mahaalyahya crosslingualtransferlearningforarabictaskorienteddialoguesystemsusingmultilingualtransformermodelmt5 |