A two-stage fine-tuning method for low-resource cross-lingual summarization
Cross-lingual summarization (CLS) is the task of condensing lengthy source language text into a concise summary in a target language. This presents a dual challenge, demanding both cross-language semantic understanding (i.e., semantic alignment) and effective information compression capabilities. Tr...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
AIMS Press
2024-01-01
|
| Series: | Mathematical Biosciences and Engineering |
| Subjects: | |
| Online Access: | https://aimspress.com/article/doi/10.3934/mbe.2024047?viewType=HTML |
| _version_ | 1797338458111344640 |
|---|---|
| author | Kaixiong Zhang Yongbing Zhang Zhengtao Yu Yuxin Huang Kaiwen Tan |
| author_facet | Kaixiong Zhang Yongbing Zhang Zhengtao Yu Yuxin Huang Kaiwen Tan |
| author_sort | Kaixiong Zhang |
| collection | DOAJ |
| description | Cross-lingual summarization (CLS) is the task of condensing lengthy source language text into a concise summary in a target language. This presents a dual challenge, demanding both cross-language semantic understanding (i.e., semantic alignment) and effective information compression capabilities. Traditionally, researchers have tackled these challenges using two types of methods: pipeline methods (e.g., translate-then-summarize) and end-to-end methods. The former is intuitive but prone to error propagation, particularly for low-resource languages. The later has shown an impressive performance, due to multilingual pre-trained models (mPTMs). However, mPTMs (e.g., mBART) are primarily trained on resource-rich languages, thereby limiting their semantic alignment capabilities for low-resource languages. To address these issues, this paper integrates the intuitiveness of pipeline methods and the effectiveness of mPTMs, and then proposes a two-stage fine-tuning method for low-resource cross-lingual summarization (TFLCLS). In the first stage, by recognizing the deficiency in the semantic alignment for low-resource languages in mPTMs, a semantic alignment fine-tuning method is employed to enhance the mPTMs' understanding of such languages. In the second stage, while considering that mPTMs are not originally tailored for information compression and CLS demands the model to simultaneously align and compress, an adaptive joint fine-tuning method is introduced. This method further enhances the semantic alignment and information compression abilities of mPTMs that were trained in the first stage. To evaluate the performance of TFLCLS, a low-resource CLS dataset, named Vi2ZhLow, is constructed from scratch; moreover, two additional low-resource CLS datasets, En2ZhLow and Zh2EnLow, are synthesized from widely used large-scale CLS datasets. Experimental results show that TFCLS outperforms state-of-the-art methods by 18.88%, 12.71% and 16.91% in ROUGE-2 on the three datasets, respectively, even when limited with only 5,000 training samples. |
| first_indexed | 2024-03-08T09:31:31Z |
| format | Article |
| id | doaj.art-f583b35157944fc4a16b198abb24dcc8 |
| institution | Directory Open Access Journal |
| issn | 1551-0018 |
| language | English |
| last_indexed | 2024-03-08T09:31:31Z |
| publishDate | 2024-01-01 |
| publisher | AIMS Press |
| record_format | Article |
| series | Mathematical Biosciences and Engineering |
| spelling | doaj.art-f583b35157944fc4a16b198abb24dcc82024-01-31T01:44:06ZengAIMS PressMathematical Biosciences and Engineering1551-00182024-01-012111125114310.3934/mbe.2024047A two-stage fine-tuning method for low-resource cross-lingual summarizationKaixiong Zhang0Yongbing Zhang 1Zhengtao Yu2Yuxin Huang3Kaiwen Tan41. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming 650500, China1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming 650500, China1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming 650500, China1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming 650500, China1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology, Kunming 650500, ChinaCross-lingual summarization (CLS) is the task of condensing lengthy source language text into a concise summary in a target language. This presents a dual challenge, demanding both cross-language semantic understanding (i.e., semantic alignment) and effective information compression capabilities. Traditionally, researchers have tackled these challenges using two types of methods: pipeline methods (e.g., translate-then-summarize) and end-to-end methods. The former is intuitive but prone to error propagation, particularly for low-resource languages. The later has shown an impressive performance, due to multilingual pre-trained models (mPTMs). However, mPTMs (e.g., mBART) are primarily trained on resource-rich languages, thereby limiting their semantic alignment capabilities for low-resource languages. To address these issues, this paper integrates the intuitiveness of pipeline methods and the effectiveness of mPTMs, and then proposes a two-stage fine-tuning method for low-resource cross-lingual summarization (TFLCLS). In the first stage, by recognizing the deficiency in the semantic alignment for low-resource languages in mPTMs, a semantic alignment fine-tuning method is employed to enhance the mPTMs' understanding of such languages. In the second stage, while considering that mPTMs are not originally tailored for information compression and CLS demands the model to simultaneously align and compress, an adaptive joint fine-tuning method is introduced. This method further enhances the semantic alignment and information compression abilities of mPTMs that were trained in the first stage. To evaluate the performance of TFLCLS, a low-resource CLS dataset, named Vi2ZhLow, is constructed from scratch; moreover, two additional low-resource CLS datasets, En2ZhLow and Zh2EnLow, are synthesized from widely used large-scale CLS datasets. Experimental results show that TFCLS outperforms state-of-the-art methods by 18.88%, 12.71% and 16.91% in ROUGE-2 on the three datasets, respectively, even when limited with only 5,000 training samples.https://aimspress.com/article/doi/10.3934/mbe.2024047?viewType=HTMLcross-linguallow-resourcesummarizationfine-tuning |
| spellingShingle | Kaixiong Zhang Yongbing Zhang Zhengtao Yu Yuxin Huang Kaiwen Tan A two-stage fine-tuning method for low-resource cross-lingual summarization Mathematical Biosciences and Engineering cross-lingual low-resource summarization fine-tuning |
| title | A two-stage fine-tuning method for low-resource cross-lingual summarization |
| title_full | A two-stage fine-tuning method for low-resource cross-lingual summarization |
| title_fullStr | A two-stage fine-tuning method for low-resource cross-lingual summarization |
| title_full_unstemmed | A two-stage fine-tuning method for low-resource cross-lingual summarization |
| title_short | A two-stage fine-tuning method for low-resource cross-lingual summarization |
| title_sort | two stage fine tuning method for low resource cross lingual summarization |
| topic | cross-lingual low-resource summarization fine-tuning |
| url | https://aimspress.com/article/doi/10.3934/mbe.2024047?viewType=HTML |
| work_keys_str_mv | AT kaixiongzhang atwostagefinetuningmethodforlowresourcecrosslingualsummarization AT yongbingzhang atwostagefinetuningmethodforlowresourcecrosslingualsummarization AT zhengtaoyu atwostagefinetuningmethodforlowresourcecrosslingualsummarization AT yuxinhuang atwostagefinetuningmethodforlowresourcecrosslingualsummarization AT kaiwentan atwostagefinetuningmethodforlowresourcecrosslingualsummarization AT kaixiongzhang twostagefinetuningmethodforlowresourcecrosslingualsummarization AT yongbingzhang twostagefinetuningmethodforlowresourcecrosslingualsummarization AT zhengtaoyu twostagefinetuningmethodforlowresourcecrosslingualsummarization AT yuxinhuang twostagefinetuningmethodforlowresourcecrosslingualsummarization AT kaiwentan twostagefinetuningmethodforlowresourcecrosslingualsummarization |