Effective Transfer Learning with Label-Based Discriminative Feature Learning
The performance of natural language processing with a transfer learning methodology has improved by applying pre-training language models to downstream tasks with a large number of general data. However, because the data used in pre-training are irrelevant to the downstream tasks, a problem occurs i...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/5/2025 |
| _version_ | 1827650551803805696 |
|---|---|
| author | Gyunyeop Kim Sangwoo Kang |
| author_facet | Gyunyeop Kim Sangwoo Kang |
| author_sort | Gyunyeop Kim |
| collection | DOAJ |
| description | The performance of natural language processing with a transfer learning methodology has improved by applying pre-training language models to downstream tasks with a large number of general data. However, because the data used in pre-training are irrelevant to the downstream tasks, a problem occurs in that it learns general features rather than those features specific to the downstream tasks. In this paper, a novel learning method is proposed for embedding pre-trained models to learn specific features of such tasks. The proposed method learns the label features of downstream tasks through contrast learning using label embedding and sampled data pairs. To demonstrate the performance of the proposed method, we conducted experiments on sentence classification datasets and evaluated whether the features of the downstream tasks have been learned through a PCA and a clustering of the embeddings. |
| first_indexed | 2024-03-09T20:20:41Z |
| format | Article |
| id | doaj.art-b129c414ecc94817a82807437eb4b9ac |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T20:20:41Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-b129c414ecc94817a82807437eb4b9ac2023-11-23T23:49:58ZengMDPI AGSensors1424-82202022-03-01225202510.3390/s22052025Effective Transfer Learning with Label-Based Discriminative Feature LearningGyunyeop Kim0Sangwoo Kang1School of Computing, Gachon University, Seongnam 13120, KoreaSchool of Computing, Gachon University, Seongnam 13120, KoreaThe performance of natural language processing with a transfer learning methodology has improved by applying pre-training language models to downstream tasks with a large number of general data. However, because the data used in pre-training are irrelevant to the downstream tasks, a problem occurs in that it learns general features rather than those features specific to the downstream tasks. In this paper, a novel learning method is proposed for embedding pre-trained models to learn specific features of such tasks. The proposed method learns the label features of downstream tasks through contrast learning using label embedding and sampled data pairs. To demonstrate the performance of the proposed method, we conducted experiments on sentence classification datasets and evaluated whether the features of the downstream tasks have been learned through a PCA and a clustering of the embeddings.https://www.mdpi.com/1424-8220/22/5/2025natural language processingtransfer learningpre-trainingword embedding |
| spellingShingle | Gyunyeop Kim Sangwoo Kang Effective Transfer Learning with Label-Based Discriminative Feature Learning Sensors natural language processing transfer learning pre-training word embedding |
| title | Effective Transfer Learning with Label-Based Discriminative Feature Learning |
| title_full | Effective Transfer Learning with Label-Based Discriminative Feature Learning |
| title_fullStr | Effective Transfer Learning with Label-Based Discriminative Feature Learning |
| title_full_unstemmed | Effective Transfer Learning with Label-Based Discriminative Feature Learning |
| title_short | Effective Transfer Learning with Label-Based Discriminative Feature Learning |
| title_sort | effective transfer learning with label based discriminative feature learning |
| topic | natural language processing transfer learning pre-training word embedding |
| url | https://www.mdpi.com/1424-8220/22/5/2025 |
| work_keys_str_mv | AT gyunyeopkim effectivetransferlearningwithlabelbaseddiscriminativefeaturelearning AT sangwookang effectivetransferlearningwithlabelbaseddiscriminativefeaturelearning |