Clinical prompt learning with frozen language models
When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the p...
| Main Authors: | , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
IEEE
2023
|
| _version_ | 1811139734256222208 |
|---|---|
| author | Taylor, N Zhang, Y Joyce, DW Gao, Z Kormilitzin, A Nevado-Holgado, A |
| author_facet | Taylor, N Zhang, Y Joyce, DW Gao, Z Kormilitzin, A Nevado-Holgado, A |
| author_sort | Taylor, N |
| collection | OXFORD |
| description | When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt. |
| first_indexed | 2024-03-07T08:00:42Z |
| format | Journal article |
| id | oxford-uuid:9233f479-ad86-45c2-b805-ece08844c411 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-09-25T04:10:47Z |
| publishDate | 2023 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:9233f479-ad86-45c2-b805-ece08844c4112024-06-20T09:43:01ZClinical prompt learning with frozen language modelsJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:9233f479-ad86-45c2-b805-ece08844c411EnglishSymplectic ElementsIEEE2023Taylor, NZhang, YJoyce, DWGao, ZKormilitzin, ANevado-Holgado, AWhen the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt. |
| spellingShingle | Taylor, N Zhang, Y Joyce, DW Gao, Z Kormilitzin, A Nevado-Holgado, A Clinical prompt learning with frozen language models |
| title | Clinical prompt learning with frozen language models |
| title_full | Clinical prompt learning with frozen language models |
| title_fullStr | Clinical prompt learning with frozen language models |
| title_full_unstemmed | Clinical prompt learning with frozen language models |
| title_short | Clinical prompt learning with frozen language models |
| title_sort | clinical prompt learning with frozen language models |
| work_keys_str_mv | AT taylorn clinicalpromptlearningwithfrozenlanguagemodels AT zhangy clinicalpromptlearningwithfrozenlanguagemodels AT joycedw clinicalpromptlearningwithfrozenlanguagemodels AT gaoz clinicalpromptlearningwithfrozenlanguagemodels AT kormilitzina clinicalpromptlearningwithfrozenlanguagemodels AT nevadoholgadoa clinicalpromptlearningwithfrozenlanguagemodels |