Pruning Adapters with Lottery Ticket
Massively pre-trained transformer models such as BERT have gained great success in many downstream NLP tasks. However, they are computationally expensive to fine-tune, slow for inference, and have large storage requirements. So, transfer learning with adapter modules has been introduced and has beco...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-02-01
|
| Series: | Algorithms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4893/15/2/63 |
| _version_ | 1827657744506683392 |
|---|---|
| author | Jiarun Wu Qingliang Chen |
| author_facet | Jiarun Wu Qingliang Chen |
| author_sort | Jiarun Wu |
| collection | DOAJ |
| description | Massively pre-trained transformer models such as BERT have gained great success in many downstream NLP tasks. However, they are computationally expensive to fine-tune, slow for inference, and have large storage requirements. So, transfer learning with adapter modules has been introduced and has become a remarkable solution for those problems. Nevertheless, recent studies reveal that the parameters in adapters are actually still quite redundant, which could slow down inference speed when fusing multiple adapters for a specific downstream task, and thus, they can be further reduced. To address this issue, we propose three novel ways to prune the adapter modules iteratively based on the prestigious Lottery Ticket Hypothesis. Extensive experiments on the GLUE datasets show that the pruned adapters can achieve state-of-the-art results, with sizes reduced significantly while performance remains unchanged, and some pruned adapters even outperform the ones with the same size that are fine-tuned alone without pruning. |
| first_indexed | 2024-03-09T22:48:20Z |
| format | Article |
| id | doaj.art-503028415cd84b2a87bc39f41a47dcc2 |
| institution | Directory Open Access Journal |
| issn | 1999-4893 |
| language | English |
| last_indexed | 2024-03-09T22:48:20Z |
| publishDate | 2022-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Algorithms |
| spelling | doaj.art-503028415cd84b2a87bc39f41a47dcc22023-11-23T18:24:26ZengMDPI AGAlgorithms1999-48932022-02-011526310.3390/a15020063Pruning Adapters with Lottery TicketJiarun Wu0Qingliang Chen1Department of Computer Science, Jinan University, Guangzhou 510632, ChinaDepartment of Computer Science, Jinan University, Guangzhou 510632, ChinaMassively pre-trained transformer models such as BERT have gained great success in many downstream NLP tasks. However, they are computationally expensive to fine-tune, slow for inference, and have large storage requirements. So, transfer learning with adapter modules has been introduced and has become a remarkable solution for those problems. Nevertheless, recent studies reveal that the parameters in adapters are actually still quite redundant, which could slow down inference speed when fusing multiple adapters for a specific downstream task, and thus, they can be further reduced. To address this issue, we propose three novel ways to prune the adapter modules iteratively based on the prestigious Lottery Ticket Hypothesis. Extensive experiments on the GLUE datasets show that the pruned adapters can achieve state-of-the-art results, with sizes reduced significantly while performance remains unchanged, and some pruned adapters even outperform the ones with the same size that are fine-tuned alone without pruning.https://www.mdpi.com/1999-4893/15/2/63pre-trained transformer modeladapterprune |
| spellingShingle | Jiarun Wu Qingliang Chen Pruning Adapters with Lottery Ticket Algorithms pre-trained transformer model adapter prune |
| title | Pruning Adapters with Lottery Ticket |
| title_full | Pruning Adapters with Lottery Ticket |
| title_fullStr | Pruning Adapters with Lottery Ticket |
| title_full_unstemmed | Pruning Adapters with Lottery Ticket |
| title_short | Pruning Adapters with Lottery Ticket |
| title_sort | pruning adapters with lottery ticket |
| topic | pre-trained transformer model adapter prune |
| url | https://www.mdpi.com/1999-4893/15/2/63 |
| work_keys_str_mv | AT jiarunwu pruningadapterswithlotteryticket AT qingliangchen pruningadapterswithlotteryticket |