Leveraging linguistic knowledge to enhance low-resource NLP applications
Natural Language Processing (NLP) empowers computers to process and analyze vast amounts of text data. The introduction of pre-trained language models (PLMs) has significantly advanced NLP by incorporating deep learning algorithms, thereby enhancing the handling of natural language understanding (NL...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis-Doctor of Philosophy |
| Language: | English |
| Published: |
Nanyang Technological University
2025
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/182513 |
| _version_ | 1824455223461019648 |
|---|---|
| author | Zhu, Zixiao |
| author2 | Mao Kezhi |
| author_facet | Mao Kezhi Zhu, Zixiao |
| author_sort | Zhu, Zixiao |
| collection | NTU |
| description | Natural Language Processing (NLP) empowers computers to process and analyze vast amounts of text data. The introduction of pre-trained language models (PLMs) has significantly advanced NLP by incorporating deep learning algorithms, thereby enhancing the handling of natural language understanding (NLU) tasks. However, due to their universal design, PLMs might not perform optimally in specialized tasks if essential features are not included during initial training. As a result, several training paradigms have been developed to enhance the downstream performance of PLMs, as outlined below.
Promptless fine-tuning, a common training method, adapts PLMs to specific tasks by modifying model parameters using task-specific training data. This approach has proven effective across different low-resource deep-learning models. Nevertheless, fine-tuning may face challenges such as overfitting or lack of robustness under conditions of training data scarcity. To mitigate this, the prompt-based learning paradigm is introduced, utilizing natural language prompts to enhance model understanding. Within this paradigm, fixed-prompt LM tuning wraps input sentences into a template, allowing the PLM to engage in tasks like masked language prediction or inference understanding. These techniques, which incorporate task descriptions, have shown substantial efficacy in few-shot learning. However, the need for highly comprehensive and generalizable templates presents challenges in design, and the issues of promptless fine-tuning persist, albeit mitigated, in low-data settings. With the rise of large language models (LLMs), traditional fine-tuning has become prohibitive for many users due to computational demands. As a result, tuning-free prompting has emerged as a novel approach for those with limited computational resources, leveraging the inherent language understanding capabilities of LLMs. This paradigm, including in-context learning (ICL), relies on few-shot instruction and demonstration to prompt LLM responses. The effectiveness of ICL heavily depends on how sample-label pairs are organized within demonstrations. The strategic selection and ranking of these pairs to maximize understanding with minimal data is a critical area of ongoing research in NLP applications.
External knowledge is proven to be beneficial for deep learning algorithms to reduce the reliance on training data and provide additional useful information.
How to effectively incorporate useful external knowledge to excite LM's capabilities in low-resource NLP applications remains an open research question.
This thesis investigates how incorporating existing knowledge into training paradigms can enhance NLP applications under low-resource conditions such as training data scarcity and limited computational resources. By leveraging external knowledge as prior knowledge, we aim to achieve improved text representations, more nuanced task descriptions, and richer label information instructions, thereby reducing the model's dependency on training data and enhancing its understanding capabilities. We demonstrate the advantages of integrating additional knowledge into deep learning systems and offer frameworks to apply this knowledge across different training paradigms, thereby improving performance on various NLP tasks, particularly under low-resource conditions. Specifically,
1. In the promptless fine-tuning paradigm, we first focus on fine-tuning word embeddings for task-related words, thereby enriching the conceptual knowledge available to compositional neural networks during feature learning in emotion recognition. This approach effectively enhances emotional keyword attention. We then extend this method by incorporating domain-specific lexical knowledge to improve the pre-trained word representations within a learning network, enriching the context-based word embeddings with discriminative features, providing more semantic insights, and bolstering performance across various classification tasks.
2. In the fixed-prompt LM tuning paradigm, we introduce a novel task description that incorporates dictionary knowledge to offer extensive semantic insights into labels. Building on this strategy, we devise an approach to augment few-shot classification performance within an entailment-based framework, significantly enhancing the efficiency of using limited training data and even facilitating zero-shot learning.
3. In the tuning-free prompting paradigm, we demonstrate how to incorporate label-related words into demonstrations based on LLM feedback, creating effective sample-and-label-level demonstrations. Additionally, we propose an innovative method that uses multiple-label words in demonstrations instead of traditional class names, offering more detailed and varied label instructions for LM understanding, thereby improving in-context learning (ICL) classification capabilities. |
| first_indexed | 2025-02-19T03:34:48Z |
| format | Thesis-Doctor of Philosophy |
| id | ntu-10356/182513 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:34:48Z |
| publishDate | 2025 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1825132025-02-09T15:38:56Z Leveraging linguistic knowledge to enhance low-resource NLP applications Zhu, Zixiao Mao Kezhi Interdisciplinary Graduate School (IGS) Singapore-ETH Centre Institute of Catastrophe Risk Management (ICRM) EKZMao@ntu.edu.sg Computer and Information Science Engineering Natural language processing Linguisitc knowledge enhancing Pre-trained language model Large language model Natural Language Processing (NLP) empowers computers to process and analyze vast amounts of text data. The introduction of pre-trained language models (PLMs) has significantly advanced NLP by incorporating deep learning algorithms, thereby enhancing the handling of natural language understanding (NLU) tasks. However, due to their universal design, PLMs might not perform optimally in specialized tasks if essential features are not included during initial training. As a result, several training paradigms have been developed to enhance the downstream performance of PLMs, as outlined below. Promptless fine-tuning, a common training method, adapts PLMs to specific tasks by modifying model parameters using task-specific training data. This approach has proven effective across different low-resource deep-learning models. Nevertheless, fine-tuning may face challenges such as overfitting or lack of robustness under conditions of training data scarcity. To mitigate this, the prompt-based learning paradigm is introduced, utilizing natural language prompts to enhance model understanding. Within this paradigm, fixed-prompt LM tuning wraps input sentences into a template, allowing the PLM to engage in tasks like masked language prediction or inference understanding. These techniques, which incorporate task descriptions, have shown substantial efficacy in few-shot learning. However, the need for highly comprehensive and generalizable templates presents challenges in design, and the issues of promptless fine-tuning persist, albeit mitigated, in low-data settings. With the rise of large language models (LLMs), traditional fine-tuning has become prohibitive for many users due to computational demands. As a result, tuning-free prompting has emerged as a novel approach for those with limited computational resources, leveraging the inherent language understanding capabilities of LLMs. This paradigm, including in-context learning (ICL), relies on few-shot instruction and demonstration to prompt LLM responses. The effectiveness of ICL heavily depends on how sample-label pairs are organized within demonstrations. The strategic selection and ranking of these pairs to maximize understanding with minimal data is a critical area of ongoing research in NLP applications. External knowledge is proven to be beneficial for deep learning algorithms to reduce the reliance on training data and provide additional useful information. How to effectively incorporate useful external knowledge to excite LM's capabilities in low-resource NLP applications remains an open research question. This thesis investigates how incorporating existing knowledge into training paradigms can enhance NLP applications under low-resource conditions such as training data scarcity and limited computational resources. By leveraging external knowledge as prior knowledge, we aim to achieve improved text representations, more nuanced task descriptions, and richer label information instructions, thereby reducing the model's dependency on training data and enhancing its understanding capabilities. We demonstrate the advantages of integrating additional knowledge into deep learning systems and offer frameworks to apply this knowledge across different training paradigms, thereby improving performance on various NLP tasks, particularly under low-resource conditions. Specifically, 1. In the promptless fine-tuning paradigm, we first focus on fine-tuning word embeddings for task-related words, thereby enriching the conceptual knowledge available to compositional neural networks during feature learning in emotion recognition. This approach effectively enhances emotional keyword attention. We then extend this method by incorporating domain-specific lexical knowledge to improve the pre-trained word representations within a learning network, enriching the context-based word embeddings with discriminative features, providing more semantic insights, and bolstering performance across various classification tasks. 2. In the fixed-prompt LM tuning paradigm, we introduce a novel task description that incorporates dictionary knowledge to offer extensive semantic insights into labels. Building on this strategy, we devise an approach to augment few-shot classification performance within an entailment-based framework, significantly enhancing the efficiency of using limited training data and even facilitating zero-shot learning. 3. In the tuning-free prompting paradigm, we demonstrate how to incorporate label-related words into demonstrations based on LLM feedback, creating effective sample-and-label-level demonstrations. Additionally, we propose an innovative method that uses multiple-label words in demonstrations instead of traditional class names, offering more detailed and varied label instructions for LM understanding, thereby improving in-context learning (ICL) classification capabilities. Doctor of Philosophy 2025-02-06T01:08:20Z 2025-02-06T01:08:20Z 2025 Thesis-Doctor of Philosophy Zhu, Z. (2025). Leveraging linguistic knowledge to enhance low-resource NLP applications. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/182513 https://hdl.handle.net/10356/182513 en This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). application/pdf Nanyang Technological University |
| spellingShingle | Computer and Information Science Engineering Natural language processing Linguisitc knowledge enhancing Pre-trained language model Large language model Zhu, Zixiao Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title | Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title_full | Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title_fullStr | Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title_full_unstemmed | Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title_short | Leveraging linguistic knowledge to enhance low-resource NLP applications |
| title_sort | leveraging linguistic knowledge to enhance low resource nlp applications |
| topic | Computer and Information Science Engineering Natural language processing Linguisitc knowledge enhancing Pre-trained language model Large language model |
| url | https://hdl.handle.net/10356/182513 |
| work_keys_str_mv | AT zhuzixiao leveraginglinguisticknowledgetoenhancelowresourcenlpapplications |