Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation
To achieve lifelong language learning, pseudo-rehearsal methods leverage samples generated from a language model to refresh the knowledge of previously learned tasks. Without proper controls, however, these methods could fail to retain the knowledge of complex tasks with longer texts since most of t...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2022-06-01
|
| Series: | Computational Linguistics |
| Online Access: | http://dx.doi.org/10.1162/coli_a_00449 |
| _version_ | 1797795428886904832 |
|---|---|
| author | Kasidis Kanwatchara Thanapapas Horsuwan Piyawat Lertvittayakumjorn Boonserm Kijsirikul Peerapon Vateekul |
| author_facet | Kasidis Kanwatchara Thanapapas Horsuwan Piyawat Lertvittayakumjorn Boonserm Kijsirikul Peerapon Vateekul |
| author_sort | Kasidis Kanwatchara |
| collection | DOAJ |
| description | To achieve lifelong language learning, pseudo-rehearsal methods leverage samples generated from a language model to refresh the knowledge of previously learned tasks. Without proper controls, however, these methods could fail to retain the knowledge of complex tasks with longer texts since most of the generated samples are low in quality. To overcome the problem, we propose three specific contributions. First, we utilize double language models, each of which specializes in a specific part of the input, to produce high-quality pseudo samples. Second, we reduce the number of parameters used by applying adapter modules to enhance training efficiency. Third, we further improve the overall quality of pseudo samples using temporal ensembling and sample regeneration. The results show that our framework achieves significant improvement over baselines on multiple task sequences. Also, our pseudo sample analysis reveals
helpful insights for designing even better pseudo-rehearsal methods in the future. |
| first_indexed | 2024-03-13T03:17:48Z |
| format | Article |
| id | doaj.art-144d323fe8a146849b81ca4f653a6e7a |
| institution | Directory Open Access Journal |
| issn | 1530-9312 |
| language | English |
| last_indexed | 2024-03-13T03:17:48Z |
| publishDate | 2022-06-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Computational Linguistics |
| spelling | doaj.art-144d323fe8a146849b81ca4f653a6e7a2023-06-25T14:50:05ZengThe MIT PressComputational Linguistics1530-93122022-06-0148410.1162/coli_a_00449Enhancing Lifelong Language Learning by Improving Pseudo-Sample GenerationKasidis KanwatcharaThanapapas HorsuwanPiyawat LertvittayakumjornBoonserm KijsirikulPeerapon VateekulTo achieve lifelong language learning, pseudo-rehearsal methods leverage samples generated from a language model to refresh the knowledge of previously learned tasks. Without proper controls, however, these methods could fail to retain the knowledge of complex tasks with longer texts since most of the generated samples are low in quality. To overcome the problem, we propose three specific contributions. First, we utilize double language models, each of which specializes in a specific part of the input, to produce high-quality pseudo samples. Second, we reduce the number of parameters used by applying adapter modules to enhance training efficiency. Third, we further improve the overall quality of pseudo samples using temporal ensembling and sample regeneration. The results show that our framework achieves significant improvement over baselines on multiple task sequences. Also, our pseudo sample analysis reveals helpful insights for designing even better pseudo-rehearsal methods in the future.http://dx.doi.org/10.1162/coli_a_00449 |
| spellingShingle | Kasidis Kanwatchara Thanapapas Horsuwan Piyawat Lertvittayakumjorn Boonserm Kijsirikul Peerapon Vateekul Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation Computational Linguistics |
| title | Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation |
| title_full | Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation |
| title_fullStr | Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation |
| title_full_unstemmed | Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation |
| title_short | Enhancing Lifelong Language Learning by Improving Pseudo-Sample Generation |
| title_sort | enhancing lifelong language learning by improving pseudo sample generation |
| url | http://dx.doi.org/10.1162/coli_a_00449 |
| work_keys_str_mv | AT kasidiskanwatchara enhancinglifelonglanguagelearningbyimprovingpseudosamplegeneration AT thanapapashorsuwan enhancinglifelonglanguagelearningbyimprovingpseudosamplegeneration AT piyawatlertvittayakumjorn enhancinglifelonglanguagelearningbyimprovingpseudosamplegeneration AT boonsermkijsirikul enhancinglifelonglanguagelearningbyimprovingpseudosamplegeneration AT peeraponvateekul enhancinglifelonglanguagelearningbyimprovingpseudosamplegeneration |