Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism
There is a need to extract meaningful information from big data, classify it into different categories, and predict end-user behavior or emotions. Large amounts of data are generated from various sources such as social media and websites. Text classification is a representative research topic in the...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-08-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/10/17/5841 |
| _version_ | 1797556002863710208 |
|---|---|
| author | Beakcheol Jang Myeonghwi Kim Gaspard Harerimana Sang-ug Kang Jong Wook Kim |
| author_facet | Beakcheol Jang Myeonghwi Kim Gaspard Harerimana Sang-ug Kang Jong Wook Kim |
| author_sort | Beakcheol Jang |
| collection | DOAJ |
| description | There is a need to extract meaningful information from big data, classify it into different categories, and predict end-user behavior or emotions. Large amounts of data are generated from various sources such as social media and websites. Text classification is a representative research topic in the field of natural-language processing that categorizes unstructured text data into meaningful categorical classes. The long short-term memory (LSTM) model and the convolutional neural network for sentence classification produce accurate results and have been recently used in various natural-language processing (NLP) tasks. Convolutional neural network (CNN) models use convolutional layers and maximum pooling or max-overtime pooling layers to extract higher-level features, while LSTM models can capture long-term dependencies between word sequences hence are better used for text classification. However, even with the hybrid approach that leverages the powers of these two deep-learning models, the number of features to remember for classification remains huge, hence hindering the training process. In this study, we propose an attention-based Bi-LSTM+CNN hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism. We trained the model using the Internet Movie Database (IMDB) movie review data to evaluate the performance of the proposed model, and the test results showed that the proposed hybrid attention Bi-LSTM+CNN model produces more accurate classification results, as well as higher recall and F1 scores, than individual multi-layer perceptron (MLP), CNN or LSTM models as well as the hybrid models. |
| first_indexed | 2024-03-10T16:56:31Z |
| format | Article |
| id | doaj.art-0ccfc145503b44d49f31704c16e96500 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T16:56:31Z |
| publishDate | 2020-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-0ccfc145503b44d49f31704c16e965002023-11-20T11:06:45ZengMDPI AGApplied Sciences2076-34172020-08-011017584110.3390/app10175841Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention MechanismBeakcheol Jang0Myeonghwi Kim1Gaspard Harerimana2Sang-ug Kang3Jong Wook Kim4Department of Computer Science, Sangmyung University, Seoul 03016, KoreaDepartment of Computer Science, Sangmyung University, Seoul 03016, KoreaDepartment of Computer Science, Sangmyung University, Seoul 03016, KoreaDepartment of Computer Science, Sangmyung University, Seoul 03016, KoreaDepartment of Computer Science, Sangmyung University, Seoul 03016, KoreaThere is a need to extract meaningful information from big data, classify it into different categories, and predict end-user behavior or emotions. Large amounts of data are generated from various sources such as social media and websites. Text classification is a representative research topic in the field of natural-language processing that categorizes unstructured text data into meaningful categorical classes. The long short-term memory (LSTM) model and the convolutional neural network for sentence classification produce accurate results and have been recently used in various natural-language processing (NLP) tasks. Convolutional neural network (CNN) models use convolutional layers and maximum pooling or max-overtime pooling layers to extract higher-level features, while LSTM models can capture long-term dependencies between word sequences hence are better used for text classification. However, even with the hybrid approach that leverages the powers of these two deep-learning models, the number of features to remember for classification remains huge, hence hindering the training process. In this study, we propose an attention-based Bi-LSTM+CNN hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism. We trained the model using the Internet Movie Database (IMDB) movie review data to evaluate the performance of the proposed model, and the test results showed that the proposed hybrid attention Bi-LSTM+CNN model produces more accurate classification results, as well as higher recall and F1 scores, than individual multi-layer perceptron (MLP), CNN or LSTM models as well as the hybrid models.https://www.mdpi.com/2076-3417/10/17/5841text classificationCNNBi-LSTMattention mechanism |
| spellingShingle | Beakcheol Jang Myeonghwi Kim Gaspard Harerimana Sang-ug Kang Jong Wook Kim Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism Applied Sciences text classification CNN Bi-LSTM attention mechanism |
| title | Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism |
| title_full | Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism |
| title_fullStr | Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism |
| title_full_unstemmed | Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism |
| title_short | Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism |
| title_sort | bi lstm model to increase accuracy in text classification combining word2vec cnn and attention mechanism |
| topic | text classification CNN Bi-LSTM attention mechanism |
| url | https://www.mdpi.com/2076-3417/10/17/5841 |
| work_keys_str_mv | AT beakcheoljang bilstmmodeltoincreaseaccuracyintextclassificationcombiningword2veccnnandattentionmechanism AT myeonghwikim bilstmmodeltoincreaseaccuracyintextclassificationcombiningword2veccnnandattentionmechanism AT gaspardharerimana bilstmmodeltoincreaseaccuracyintextclassificationcombiningword2veccnnandattentionmechanism AT sangugkang bilstmmodeltoincreaseaccuracyintextclassificationcombiningword2veccnnandattentionmechanism AT jongwookkim bilstmmodeltoincreaseaccuracyintextclassificationcombiningword2veccnnandattentionmechanism |