SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT
BERT, the most popular deep learning language model, has yielded breakthrough results in various NLP tasks. However, the semantic representation space learned by BERT has the property of anisotropy. Therefore, BERT needs to be fine-tuned for certain downstream tasks such as Semantic Textual Similari...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/3/1913 |
| _version_ | 1797625095852654592 |
|---|---|
| author | Somaiyeh Dehghan Mehmet Fatih Amasyali |
| author_facet | Somaiyeh Dehghan Mehmet Fatih Amasyali |
| author_sort | Somaiyeh Dehghan |
| collection | DOAJ |
| description | BERT, the most popular deep learning language model, has yielded breakthrough results in various NLP tasks. However, the semantic representation space learned by BERT has the property of anisotropy. Therefore, BERT needs to be fine-tuned for certain downstream tasks such as Semantic Textual Similarity (STS). To overcome this problem and improve the sentence representation space, some contrastive learning methods have been proposed for fine-tuning BERT. However, existing contrastive learning models do not consider the importance of input triplets in terms of easy and hard negatives during training. In this paper, we propose the SelfCCL: Curriculum Contrastive Learning model by Transferring Self-taught Knowledge for Fine-Tuning BERT, which mimics the two ways that humans learn about the world around them, namely contrastive learning and curriculum learning. The former learns by contrasting similar and dissimilar samples. The latter is inspired by the way humans learn from the simplest concepts to the most complex concepts. Our model also performs this training by transferring self-taught knowledge. That is, the model figures out which triplets are easy or difficult based on previously learned knowledge, and then learns based on those triplets in the order of curriculum using a contrastive objective. We apply our proposed model to the BERT and Sentence BERT(SBERT) frameworks. The evaluation results of SelfCCL on the standard STS and SentEval transfer learning tasks show that using curriculum learning together with contrastive learning increases average performance to some extent. |
| first_indexed | 2024-03-11T09:51:59Z |
| format | Article |
| id | doaj.art-5ae8691454934ff289e88bcea36326ea |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T09:51:59Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-5ae8691454934ff289e88bcea36326ea2023-11-16T16:12:09ZengMDPI AGApplied Sciences2076-34172023-02-01133191310.3390/app13031913SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERTSomaiyeh Dehghan0Mehmet Fatih Amasyali1Department of Computer Engineering, Yildiz Technical University, Istanbul 34220, TurkeyDepartment of Computer Engineering, Yildiz Technical University, Istanbul 34220, TurkeyBERT, the most popular deep learning language model, has yielded breakthrough results in various NLP tasks. However, the semantic representation space learned by BERT has the property of anisotropy. Therefore, BERT needs to be fine-tuned for certain downstream tasks such as Semantic Textual Similarity (STS). To overcome this problem and improve the sentence representation space, some contrastive learning methods have been proposed for fine-tuning BERT. However, existing contrastive learning models do not consider the importance of input triplets in terms of easy and hard negatives during training. In this paper, we propose the SelfCCL: Curriculum Contrastive Learning model by Transferring Self-taught Knowledge for Fine-Tuning BERT, which mimics the two ways that humans learn about the world around them, namely contrastive learning and curriculum learning. The former learns by contrasting similar and dissimilar samples. The latter is inspired by the way humans learn from the simplest concepts to the most complex concepts. Our model also performs this training by transferring self-taught knowledge. That is, the model figures out which triplets are easy or difficult based on previously learned knowledge, and then learns based on those triplets in the order of curriculum using a contrastive objective. We apply our proposed model to the BERT and Sentence BERT(SBERT) frameworks. The evaluation results of SelfCCL on the standard STS and SentEval transfer learning tasks show that using curriculum learning together with contrastive learning increases average performance to some extent.https://www.mdpi.com/2076-3417/13/3/1913transfer learningcurriculum learningcontrastive learningself-taught learningsentence embeddingnatural language processing |
| spellingShingle | Somaiyeh Dehghan Mehmet Fatih Amasyali SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT Applied Sciences transfer learning curriculum learning contrastive learning self-taught learning sentence embedding natural language processing |
| title | SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT |
| title_full | SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT |
| title_fullStr | SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT |
| title_full_unstemmed | SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT |
| title_short | SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT |
| title_sort | selfccl curriculum contrastive learning by transferring self taught knowledge for fine tuning bert |
| topic | transfer learning curriculum learning contrastive learning self-taught learning sentence embedding natural language processing |
| url | https://www.mdpi.com/2076-3417/13/3/1913 |
| work_keys_str_mv | AT somaiyehdehghan selfcclcurriculumcontrastivelearningbytransferringselftaughtknowledgeforfinetuningbert AT mehmetfatihamasyali selfcclcurriculumcontrastivelearningbytransferringselftaughtknowledgeforfinetuningbert |