VPN: Variation on Prompt Tuning for Named-Entity Recognition
Recently, prompt-based methods have achieved a promising performance in many natural language processing benchmarks. Despite success in sentence-level classification tasks, prompt-based methods work poorly in token-level tasks, such as named entity recognition (NER), due to the sophisticated design...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/14/8359 |
| _version_ | 1797590349865025536 |
|---|---|
| author | Niu Hu Xuan Zhou Bing Xu Hanqing Liu Xiangjin Xie Hai-Tao Zheng |
| author_facet | Niu Hu Xuan Zhou Bing Xu Hanqing Liu Xiangjin Xie Hai-Tao Zheng |
| author_sort | Niu Hu |
| collection | DOAJ |
| description | Recently, prompt-based methods have achieved a promising performance in many natural language processing benchmarks. Despite success in sentence-level classification tasks, prompt-based methods work poorly in token-level tasks, such as named entity recognition (NER), due to the sophisticated design of entity-related templates. Note that the nature of prompt tuning makes full use of the parameters of the mask language model (MLM) head, while previous methods solely utilized the last hidden layer of language models (LMs) and the power of the MLM head is overlooked. In this work, we discovered the characteristics of semantic feature changes in samples after being processed using MLMs. Based on this characteristic, we designed a prompt-tuning variant for NER tasks. We let the pre-trained model predict the label words derived from the training dataset at each position and fed the generated logits (non-normalized probability) to the CRF layer. We evaluated our method on three popular datasets, and the experiments showed that our proposed method outperforms the state-of-the-art model in all three Chinese datasets. |
| first_indexed | 2024-03-11T01:19:13Z |
| format | Article |
| id | doaj.art-0e8431517258472c8df832bc73f1a8d7 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T01:19:13Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-0e8431517258472c8df832bc73f1a8d72023-11-18T18:11:58ZengMDPI AGApplied Sciences2076-34172023-07-011314835910.3390/app13148359VPN: Variation on Prompt Tuning for Named-Entity RecognitionNiu Hu0Xuan Zhou1Bing Xu2Hanqing Liu3Xiangjin Xie4Hai-Tao Zheng5Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, ChinaPAII Inc., Palo Alto, CA 94306, USAPing An Technology (Shenzhen) Co., Ltd., Shenzhen 518063, ChinaShenzhen International Graduate School, Tsinghua University, Shenzhen 518055, ChinaShenzhen International Graduate School, Tsinghua University, Shenzhen 518055, ChinaShenzhen International Graduate School, Tsinghua University, Shenzhen 518055, ChinaRecently, prompt-based methods have achieved a promising performance in many natural language processing benchmarks. Despite success in sentence-level classification tasks, prompt-based methods work poorly in token-level tasks, such as named entity recognition (NER), due to the sophisticated design of entity-related templates. Note that the nature of prompt tuning makes full use of the parameters of the mask language model (MLM) head, while previous methods solely utilized the last hidden layer of language models (LMs) and the power of the MLM head is overlooked. In this work, we discovered the characteristics of semantic feature changes in samples after being processed using MLMs. Based on this characteristic, we designed a prompt-tuning variant for NER tasks. We let the pre-trained model predict the label words derived from the training dataset at each position and fed the generated logits (non-normalized probability) to the CRF layer. We evaluated our method on three popular datasets, and the experiments showed that our proposed method outperforms the state-of-the-art model in all three Chinese datasets.https://www.mdpi.com/2076-3417/13/14/8359prompt tuningMLM headNER |
| spellingShingle | Niu Hu Xuan Zhou Bing Xu Hanqing Liu Xiangjin Xie Hai-Tao Zheng VPN: Variation on Prompt Tuning for Named-Entity Recognition Applied Sciences prompt tuning MLM head NER |
| title | VPN: Variation on Prompt Tuning for Named-Entity Recognition |
| title_full | VPN: Variation on Prompt Tuning for Named-Entity Recognition |
| title_fullStr | VPN: Variation on Prompt Tuning for Named-Entity Recognition |
| title_full_unstemmed | VPN: Variation on Prompt Tuning for Named-Entity Recognition |
| title_short | VPN: Variation on Prompt Tuning for Named-Entity Recognition |
| title_sort | vpn variation on prompt tuning for named entity recognition |
| topic | prompt tuning MLM head NER |
| url | https://www.mdpi.com/2076-3417/13/14/8359 |
| work_keys_str_mv | AT niuhu vpnvariationonprompttuningfornamedentityrecognition AT xuanzhou vpnvariationonprompttuningfornamedentityrecognition AT bingxu vpnvariationonprompttuningfornamedentityrecognition AT hanqingliu vpnvariationonprompttuningfornamedentityrecognition AT xiangjinxie vpnvariationonprompttuningfornamedentityrecognition AT haitaozheng vpnvariationonprompttuningfornamedentityrecognition |