A Joint Learning Framework With BERT for Spoken Language Understanding
Intent classification and slot filling are two essential tasks for spoken language understanding. Recently, joint learning has been shown to be effective for the two tasks. However, most joint learning methods only consider joint learning using shared parameters on the surface level rather than the...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2019-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8907842/ |
| _version_ | 1818875907599761408 |
|---|---|
| author | Zhichang Zhang Zhenwen Zhang Haoyuan Chen Zhiman Zhang |
| author_facet | Zhichang Zhang Zhenwen Zhang Haoyuan Chen Zhiman Zhang |
| author_sort | Zhichang Zhang |
| collection | DOAJ |
| description | Intent classification and slot filling are two essential tasks for spoken language understanding. Recently, joint learning has been shown to be effective for the two tasks. However, most joint learning methods only consider joint learning using shared parameters on the surface level rather than the semantic level, and these methods suffer from small-scale human-labeled training data, resulting in poor generalization capabilities, especially for rare words. In this paper, we propose a novel encoder-decoder framework based multi-task learning model, which conducts joint training for intent classification and slot filling tasks. For the encoder of our model, we encode the input sequence as context representations using bidirectional encoder representation from transformers (BERT). For the decoder, we implement two-stage decoder process in our model. In the first stage, we use an intent classification decoder to detect the user's intent. In the second stage, we leverage the intent contextual information into the slot filling decoder to predict the semantic concept tags for each word. We conduct experiments on three popular benchmark datasets: ATIS, Snips and Facebook multilingual task-oriented datasets. The experimental results show that our proposed model outperforms the state-of-the-art approaches and achieves new state-of-the-art results on both three datasets. |
| first_indexed | 2024-12-19T13:33:57Z |
| format | Article |
| id | doaj.art-298817d215c94278a1d3c581cbc5b5ef |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-19T13:33:57Z |
| publishDate | 2019-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-298817d215c94278a1d3c581cbc5b5ef2022-12-21T20:19:16ZengIEEEIEEE Access2169-35362019-01-01716884916885810.1109/ACCESS.2019.29547668907842A Joint Learning Framework With BERT for Spoken Language UnderstandingZhichang Zhang0https://orcid.org/0000-0003-3306-8493Zhenwen Zhang1https://orcid.org/0000-0001-8443-8779Haoyuan Chen2https://orcid.org/0000-0003-3120-931XZhiman Zhang3https://orcid.org/0000-0002-0993-4323College of Computer Science and Engineering, Northwest Normal University, Lanzhou, ChinaCollege of Computer Science and Engineering, Northwest Normal University, Lanzhou, ChinaCollege of Computer Science and Engineering, Northwest Normal University, Lanzhou, ChinaCollege of Computer Science and Engineering, Northwest Normal University, Lanzhou, ChinaIntent classification and slot filling are two essential tasks for spoken language understanding. Recently, joint learning has been shown to be effective for the two tasks. However, most joint learning methods only consider joint learning using shared parameters on the surface level rather than the semantic level, and these methods suffer from small-scale human-labeled training data, resulting in poor generalization capabilities, especially for rare words. In this paper, we propose a novel encoder-decoder framework based multi-task learning model, which conducts joint training for intent classification and slot filling tasks. For the encoder of our model, we encode the input sequence as context representations using bidirectional encoder representation from transformers (BERT). For the decoder, we implement two-stage decoder process in our model. In the first stage, we use an intent classification decoder to detect the user's intent. In the second stage, we leverage the intent contextual information into the slot filling decoder to predict the semantic concept tags for each word. We conduct experiments on three popular benchmark datasets: ATIS, Snips and Facebook multilingual task-oriented datasets. The experimental results show that our proposed model outperforms the state-of-the-art approaches and achieves new state-of-the-art results on both three datasets.https://ieeexplore.ieee.org/document/8907842/Spoken language understandingintent classification and slot fillingjoint learningintent-augmented mechanismpre-trained language model |
| spellingShingle | Zhichang Zhang Zhenwen Zhang Haoyuan Chen Zhiman Zhang A Joint Learning Framework With BERT for Spoken Language Understanding IEEE Access Spoken language understanding intent classification and slot filling joint learning intent-augmented mechanism pre-trained language model |
| title | A Joint Learning Framework With BERT for Spoken Language Understanding |
| title_full | A Joint Learning Framework With BERT for Spoken Language Understanding |
| title_fullStr | A Joint Learning Framework With BERT for Spoken Language Understanding |
| title_full_unstemmed | A Joint Learning Framework With BERT for Spoken Language Understanding |
| title_short | A Joint Learning Framework With BERT for Spoken Language Understanding |
| title_sort | joint learning framework with bert for spoken language understanding |
| topic | Spoken language understanding intent classification and slot filling joint learning intent-augmented mechanism pre-trained language model |
| url | https://ieeexplore.ieee.org/document/8907842/ |
| work_keys_str_mv | AT zhichangzhang ajointlearningframeworkwithbertforspokenlanguageunderstanding AT zhenwenzhang ajointlearningframeworkwithbertforspokenlanguageunderstanding AT haoyuanchen ajointlearningframeworkwithbertforspokenlanguageunderstanding AT zhimanzhang ajointlearningframeworkwithbertforspokenlanguageunderstanding AT zhichangzhang jointlearningframeworkwithbertforspokenlanguageunderstanding AT zhenwenzhang jointlearningframeworkwithbertforspokenlanguageunderstanding AT haoyuanchen jointlearningframeworkwithbertforspokenlanguageunderstanding AT zhimanzhang jointlearningframeworkwithbertforspokenlanguageunderstanding |