Pre-training model based on the transfer learning in natural language processing
Transfer learning is to apply knowledge or patterns learned in a particular field or task to different but related areas or problem. It is very prominent in terms of scarcity of data and heterogeneity of domain distribution. In the field of natural language processing, transfer learning is embodied...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | English |
| Published: |
2019
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/10356/78688 |
| _version_ | 1811682101242626048 |
|---|---|
| author | Tang, Jiayi |
| author2 | Mao Kezhi |
| author_facet | Mao Kezhi Tang, Jiayi |
| author_sort | Tang, Jiayi |
| collection | NTU |
| description | Transfer learning is to apply knowledge or patterns learned in a particular field or task to different but related areas or problem. It is very prominent in terms of scarcity of data and heterogeneity of domain distribution. In the field of natural language processing, transfer learning is embodied in the pre-training model. There are two existing strategies for applying pre-trained language representations to downstream tasks: feature-based (ELMO) and fine-tuning (GPT、BERT).
In 2018, Google released a large-scale pre-training language model BERT, which stands for Bidirectional Encoder Representations from Transformer. Compared with other pre-training model ELMO and GPT, and the classical model CNN, BERT is the latest and best-performing model up until now. Its highlights are (1) Bidirectional Transformer (2) Mask-Language Model (3) Next Sentence Prediction (4) A more general input layer and output layer. BERT model can efficiently learn text information and apply it to various NLP tasks.
In this report, we use the BERT model in two way. The first is to use the pre-training model released by Google directly and then pass the fine-tuning stage. The second is to use the BERT-as-service to use the BERT model as a sentence encode followed by a DNN classifier. Then we horizontally compare BERT with ELMO and GPT; then vertically compare BERT with different parameters. |
| first_indexed | 2024-10-01T03:51:29Z |
| format | Thesis |
| id | ntu-10356/78688 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T03:51:29Z |
| publishDate | 2019 |
| record_format | dspace |
| spelling | ntu-10356/786882023-07-04T16:08:11Z Pre-training model based on the transfer learning in natural language processing Tang, Jiayi Mao Kezhi School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Transfer learning is to apply knowledge or patterns learned in a particular field or task to different but related areas or problem. It is very prominent in terms of scarcity of data and heterogeneity of domain distribution. In the field of natural language processing, transfer learning is embodied in the pre-training model. There are two existing strategies for applying pre-trained language representations to downstream tasks: feature-based (ELMO) and fine-tuning (GPT、BERT). In 2018, Google released a large-scale pre-training language model BERT, which stands for Bidirectional Encoder Representations from Transformer. Compared with other pre-training model ELMO and GPT, and the classical model CNN, BERT is the latest and best-performing model up until now. Its highlights are (1) Bidirectional Transformer (2) Mask-Language Model (3) Next Sentence Prediction (4) A more general input layer and output layer. BERT model can efficiently learn text information and apply it to various NLP tasks. In this report, we use the BERT model in two way. The first is to use the pre-training model released by Google directly and then pass the fine-tuning stage. The second is to use the BERT-as-service to use the BERT model as a sentence encode followed by a DNN classifier. Then we horizontally compare BERT with ELMO and GPT; then vertically compare BERT with different parameters. Master of Science (Computer Control and Automation) 2019-06-25T08:21:06Z 2019-06-25T08:21:06Z 2019 Thesis http://hdl.handle.net/10356/78688 en 73 p. application/pdf |
| spellingShingle | Engineering::Electrical and electronic engineering Tang, Jiayi Pre-training model based on the transfer learning in natural language processing |
| title | Pre-training model based on the transfer learning in natural language processing |
| title_full | Pre-training model based on the transfer learning in natural language processing |
| title_fullStr | Pre-training model based on the transfer learning in natural language processing |
| title_full_unstemmed | Pre-training model based on the transfer learning in natural language processing |
| title_short | Pre-training model based on the transfer learning in natural language processing |
| title_sort | pre training model based on the transfer learning in natural language processing |
| topic | Engineering::Electrical and electronic engineering |
| url | http://hdl.handle.net/10356/78688 |
| work_keys_str_mv | AT tangjiayi pretrainingmodelbasedonthetransferlearninginnaturallanguageprocessing |