Meta Lifelong-Learning With Selective and Task-Aware Adaptation
Meta-learning has been applied to lifelong language learning due to its ability to find an optimal model for efficient adaptation to any learned tasks. Generally, meta lifelong-learning partially stores samples from seen tasks in a memory and selects some of them to train the model, refresh the know...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10433528/ |
| _version_ | 1797243352811307008 |
|---|---|
| author | Thanapapas Horsuwan Piyawat Lertvittayakumjorn Kasidis Kanwatchara Boonserm Kijsirikul Peerapon Vateekul |
| author_facet | Thanapapas Horsuwan Piyawat Lertvittayakumjorn Kasidis Kanwatchara Boonserm Kijsirikul Peerapon Vateekul |
| author_sort | Thanapapas Horsuwan |
| collection | DOAJ |
| description | Meta-learning has been applied to lifelong language learning due to its ability to find an optimal model for efficient adaptation to any learned tasks. Generally, meta lifelong-learning partially stores samples from seen tasks in a memory and selects some of them to train the model, refresh the knowledge, and adapt the model for inference. However, the sample selection for these steps was usually done in a sub-optimal manner in existing work. Hence, we propose MeLSTA (Meta Lifelong-Learning with Selective and Task-Aware Adaptation) to effectively select the samples based on task identifiers and the adaptation scores reflecting the model behavior after adaptation. The results show that MeLSTA enhances the accuracy by 1.2% over the state-of-the-art while significantly shrinking the training duration by over 6 times. Additionally, our in-depth analysis reveals the strengths and limitations of MeLSTA and existing work, providing useful insights for future designs of meta lifelong-learning for NLP. |
| first_indexed | 2024-04-24T18:53:45Z |
| format | Article |
| id | doaj.art-289dbfd7474049c3983689a1b92778fa |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-24T18:53:45Z |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-289dbfd7474049c3983689a1b92778fa2024-03-26T17:46:42ZengIEEEIEEE Access2169-35362024-01-0112340993411510.1109/ACCESS.2024.336566310433528Meta Lifelong-Learning With Selective and Task-Aware AdaptationThanapapas Horsuwan0https://orcid.org/0000-0003-4078-2660Piyawat Lertvittayakumjorn1https://orcid.org/0000-0002-2784-9827Kasidis Kanwatchara2https://orcid.org/0000-0003-3296-7331Boonserm Kijsirikul3https://orcid.org/0000-0002-9046-7151Peerapon Vateekul4https://orcid.org/0000-0001-9718-3592Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, ThailandGoogle LLC, Mountain View, CA, USADepartment of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, ThailandDepartment of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, ThailandDepartment of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, ThailandMeta-learning has been applied to lifelong language learning due to its ability to find an optimal model for efficient adaptation to any learned tasks. Generally, meta lifelong-learning partially stores samples from seen tasks in a memory and selects some of them to train the model, refresh the knowledge, and adapt the model for inference. However, the sample selection for these steps was usually done in a sub-optimal manner in existing work. Hence, we propose MeLSTA (Meta Lifelong-Learning with Selective and Task-Aware Adaptation) to effectively select the samples based on task identifiers and the adaptation scores reflecting the model behavior after adaptation. The results show that MeLSTA enhances the accuracy by 1.2% over the state-of-the-art while significantly shrinking the training duration by over 6 times. Additionally, our in-depth analysis reveals the strengths and limitations of MeLSTA and existing work, providing useful insights for future designs of meta lifelong-learning for NLP.https://ieeexplore.ieee.org/document/10433528/Catastrophic forgettinglifelong learningmeta-adaptationmeta-learningmeta lifelonglearningrelation detection |
| spellingShingle | Thanapapas Horsuwan Piyawat Lertvittayakumjorn Kasidis Kanwatchara Boonserm Kijsirikul Peerapon Vateekul Meta Lifelong-Learning With Selective and Task-Aware Adaptation IEEE Access Catastrophic forgetting lifelong learning meta-adaptation meta-learning meta lifelonglearning relation detection |
| title | Meta Lifelong-Learning With Selective and Task-Aware Adaptation |
| title_full | Meta Lifelong-Learning With Selective and Task-Aware Adaptation |
| title_fullStr | Meta Lifelong-Learning With Selective and Task-Aware Adaptation |
| title_full_unstemmed | Meta Lifelong-Learning With Selective and Task-Aware Adaptation |
| title_short | Meta Lifelong-Learning With Selective and Task-Aware Adaptation |
| title_sort | meta lifelong learning with selective and task aware adaptation |
| topic | Catastrophic forgetting lifelong learning meta-adaptation meta-learning meta lifelonglearning relation detection |
| url | https://ieeexplore.ieee.org/document/10433528/ |
| work_keys_str_mv | AT thanapapashorsuwan metalifelonglearningwithselectiveandtaskawareadaptation AT piyawatlertvittayakumjorn metalifelonglearningwithselectiveandtaskawareadaptation AT kasidiskanwatchara metalifelonglearningwithselectiveandtaskawareadaptation AT boonsermkijsirikul metalifelonglearningwithselectiveandtaskawareadaptation AT peeraponvateekul metalifelonglearningwithselectiveandtaskawareadaptation |