Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models
Text classification aims to assign predefined labels to unlabeled sentences, which tend to struggle in real-world applications when only a few annotated samples are available. Previous works generally focus on using the paradigm of meta-learning to overcome the classification difficulties brought by...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-11-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/21/11117 |
| _version_ | 1827647343897346048 |
|---|---|
| author | Chengyu Song Taihua Shao Kejing Lin Dengfeng Liu Siyuan Wang Honghui Chen |
| author_facet | Chengyu Song Taihua Shao Kejing Lin Dengfeng Liu Siyuan Wang Honghui Chen |
| author_sort | Chengyu Song |
| collection | DOAJ |
| description | Text classification aims to assign predefined labels to unlabeled sentences, which tend to struggle in real-world applications when only a few annotated samples are available. Previous works generally focus on using the paradigm of meta-learning to overcome the classification difficulties brought by insufficient data, where a set of auxiliary tasks is given. Accordingly, prompt-based approaches are proposed to deal with the low-resource issue. However, existing prompt-based methods mainly focus on English tasks, which generally apply English pretrained language models that can not directly adapt to Chinese tasks due to structural and grammatical differences. Thus, we propose a prompt-based Chinese text classification framework that uses generated natural language sequences as hints, which can alleviate the classification bottleneck well in low-resource scenarios. In detail, we first design a prompt-based fine-tuning together with a novel pipeline for automating prompt generation in Chinese. Then, we propose a refined strategy for dynamically and selectively incorporating demonstrations into each context. We present a systematic evaluation for analyzing few-shot performance on a wide range of Chinese text classification tasks. Our approach makes few assumptions about task resources and expertise and therefore constitutes a powerful, task-independent approach for few-shot learning. |
| first_indexed | 2024-03-09T19:16:49Z |
| format | Article |
| id | doaj.art-205c604de69a4f64be76dbbdf31735b8 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T19:16:49Z |
| publishDate | 2022-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-205c604de69a4f64be76dbbdf31735b82023-11-24T03:38:17ZengMDPI AGApplied Sciences2076-34172022-11-0112211111710.3390/app122111117Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language ModelsChengyu Song0Taihua Shao1Kejing Lin2Dengfeng Liu3Siyuan Wang4Honghui Chen5Science and Technology on Information Systems Engineering Laboratory, National University of Defense and Technology, No. 109 Deya Street, Changsha 410073, ChinaScience and Technology on Information Systems Engineering Laboratory, National University of Defense and Technology, No. 109 Deya Street, Changsha 410073, ChinaSchool of Information Resources Management, Renmin University of China, No. 59 Zhongguancun Street, Beijing 100872, ChinaScience and Technology on Information Systems Engineering Laboratory, National University of Defense and Technology, No. 109 Deya Street, Changsha 410073, ChinaScience and Technology on Information Systems Engineering Laboratory, National University of Defense and Technology, No. 109 Deya Street, Changsha 410073, ChinaScience and Technology on Information Systems Engineering Laboratory, National University of Defense and Technology, No. 109 Deya Street, Changsha 410073, ChinaText classification aims to assign predefined labels to unlabeled sentences, which tend to struggle in real-world applications when only a few annotated samples are available. Previous works generally focus on using the paradigm of meta-learning to overcome the classification difficulties brought by insufficient data, where a set of auxiliary tasks is given. Accordingly, prompt-based approaches are proposed to deal with the low-resource issue. However, existing prompt-based methods mainly focus on English tasks, which generally apply English pretrained language models that can not directly adapt to Chinese tasks due to structural and grammatical differences. Thus, we propose a prompt-based Chinese text classification framework that uses generated natural language sequences as hints, which can alleviate the classification bottleneck well in low-resource scenarios. In detail, we first design a prompt-based fine-tuning together with a novel pipeline for automating prompt generation in Chinese. Then, we propose a refined strategy for dynamically and selectively incorporating demonstrations into each context. We present a systematic evaluation for analyzing few-shot performance on a wide range of Chinese text classification tasks. Our approach makes few assumptions about task resources and expertise and therefore constitutes a powerful, task-independent approach for few-shot learning.https://www.mdpi.com/2076-3417/12/21/11117few-shot learningprompt learningtemplate generationdemonstration learning |
| spellingShingle | Chengyu Song Taihua Shao Kejing Lin Dengfeng Liu Siyuan Wang Honghui Chen Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models Applied Sciences few-shot learning prompt learning template generation demonstration learning |
| title | Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models |
| title_full | Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models |
| title_fullStr | Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models |
| title_full_unstemmed | Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models |
| title_short | Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models |
| title_sort | investigating prompt learning for chinese few shot text classification with pre trained language models |
| topic | few-shot learning prompt learning template generation demonstration learning |
| url | https://www.mdpi.com/2076-3417/12/21/11117 |
| work_keys_str_mv | AT chengyusong investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels AT taihuashao investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels AT kejinglin investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels AT dengfengliu investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels AT siyuanwang investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels AT honghuichen investigatingpromptlearningforchinesefewshottextclassificationwithpretrainedlanguagemodels |