A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking
The modern digital world and associated innovative and state-of-the-art applications that characterize its presence, render the current digital age a captivating era for many worldwide. These innovations include dialogue systems, such as Apple’s Siri, Google Now, and Microsoft’s Cortana, that stay o...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/3/1775 |
| _version_ | 1827760358235832320 |
|---|---|
| author | Muhammad Asif Khan Yi Huang Junlan Feng Bhuyan Kaibalya Prasad Zafar Ali Irfan Ullah Pavlos Kefalas |
| author_facet | Muhammad Asif Khan Yi Huang Junlan Feng Bhuyan Kaibalya Prasad Zafar Ali Irfan Ullah Pavlos Kefalas |
| author_sort | Muhammad Asif Khan |
| collection | DOAJ |
| description | The modern digital world and associated innovative and state-of-the-art applications that characterize its presence, render the current digital age a captivating era for many worldwide. These innovations include dialogue systems, such as Apple’s Siri, Google Now, and Microsoft’s Cortana, that stay on the personal devices of users and assist them in their daily activities. These systems track the intentions of users by analyzing their speech, context by looking at their previous turns, and several other external details, and respond or act in the form of speech output. For these systems to work efficiently, a dialogue state tracking (DST) module is required to infer the current state of the dialogue in a conversation by processing previous states up to the current state. However, developing a DST module that tracks and exploit dialogue states effectively and accurately is challenging. The notable challenges that warrant immediate attention include scalability, handling the unseen slot-value pairs during training, and retraining the model with changes in the domain ontology. In this article, we present a new end-to-end framework by combining BERT, Stacked Bidirectional LSTM (BiLSTM), and a multiple attention mechanism to formalize DST as a classification problem and address the aforementioned issues. The BERT-based module encodes the user’s and system’s utterances. The Stacked BiLSTM extracts the contextual features and multiple attention mechanisms to calculate the attention between its hidden states and the utterance embeddings. We experimentally evaluated our method against the current approaches over a variety of datasets. The results indicate a significant overall improvement. The proposed model is scalable in terms of sharing the parameters and it considers the unseen instances during training. |
| first_indexed | 2024-03-11T09:52:26Z |
| format | Article |
| id | doaj.art-ca2bf08ae1ea4feabdc7eca92d16fd4f |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T09:52:26Z |
| publishDate | 2023-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-ca2bf08ae1ea4feabdc7eca92d16fd4f2023-11-16T16:10:13ZengMDPI AGApplied Sciences2076-34172023-01-01133177510.3390/app13031775A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State TrackingMuhammad Asif Khan0Yi Huang1Junlan Feng2Bhuyan Kaibalya Prasad3Zafar Ali4Irfan Ullah5Pavlos Kefalas6School of Computer Science and Engineering, Southeast University, Nanjing 210096, ChinaChina Mobile Research Institute, Beijing 100053, ChinaChina Mobile Research Institute, Beijing 100053, ChinaDepartment of Electronics and Communication Engineering, National Institute of Technology, Rourkela 769008, IndiaSchool of Computer Science and Engineering, Southeast University, Nanjing 210096, ChinaDepartment of Computer Science, Shaheed Benazir Bhutto University, Sheringal 18050, PakistanDepartment of Informatics, Aristotle University, 54124 Thessaloniki, GreeceThe modern digital world and associated innovative and state-of-the-art applications that characterize its presence, render the current digital age a captivating era for many worldwide. These innovations include dialogue systems, such as Apple’s Siri, Google Now, and Microsoft’s Cortana, that stay on the personal devices of users and assist them in their daily activities. These systems track the intentions of users by analyzing their speech, context by looking at their previous turns, and several other external details, and respond or act in the form of speech output. For these systems to work efficiently, a dialogue state tracking (DST) module is required to infer the current state of the dialogue in a conversation by processing previous states up to the current state. However, developing a DST module that tracks and exploit dialogue states effectively and accurately is challenging. The notable challenges that warrant immediate attention include scalability, handling the unseen slot-value pairs during training, and retraining the model with changes in the domain ontology. In this article, we present a new end-to-end framework by combining BERT, Stacked Bidirectional LSTM (BiLSTM), and a multiple attention mechanism to formalize DST as a classification problem and address the aforementioned issues. The BERT-based module encodes the user’s and system’s utterances. The Stacked BiLSTM extracts the contextual features and multiple attention mechanisms to calculate the attention between its hidden states and the utterance embeddings. We experimentally evaluated our method against the current approaches over a variety of datasets. The results indicate a significant overall improvement. The proposed model is scalable in terms of sharing the parameters and it considers the unseen instances during training.https://www.mdpi.com/2076-3417/13/3/1775dialogue state trackingattention mechanismstacked BiLSTMspoken dialogue systemsBERTclassification problem |
| spellingShingle | Muhammad Asif Khan Yi Huang Junlan Feng Bhuyan Kaibalya Prasad Zafar Ali Irfan Ullah Pavlos Kefalas A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking Applied Sciences dialogue state tracking attention mechanism stacked BiLSTM spoken dialogue systems BERT classification problem |
| title | A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking |
| title_full | A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking |
| title_fullStr | A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking |
| title_full_unstemmed | A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking |
| title_short | A Multi-Attention Approach Using BERT and Stacked Bidirectional LSTM for Improved Dialogue State Tracking |
| title_sort | multi attention approach using bert and stacked bidirectional lstm for improved dialogue state tracking |
| topic | dialogue state tracking attention mechanism stacked BiLSTM spoken dialogue systems BERT classification problem |
| url | https://www.mdpi.com/2076-3417/13/3/1775 |
| work_keys_str_mv | AT muhammadasifkhan amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT yihuang amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT junlanfeng amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT bhuyankaibalyaprasad amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT zafarali amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT irfanullah amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT pavloskefalas amultiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT muhammadasifkhan multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT yihuang multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT junlanfeng multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT bhuyankaibalyaprasad multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT zafarali multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT irfanullah multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking AT pavloskefalas multiattentionapproachusingbertandstackedbidirectionallstmforimproveddialoguestatetracking |