Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding
The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavil...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9852438/ |
| _version_ | 1811274295346724864 |
|---|---|
| author | M. Kowsher Abdullah As Sami Nusrat Jahan Prottasha Mohammad Shamsul Arefin Pranab Kumar Dhar Takeshi Koshiba |
| author_facet | M. Kowsher Abdullah As Sami Nusrat Jahan Prottasha Mohammad Shamsul Arefin Pranab Kumar Dhar Takeshi Koshiba |
| author_sort | M. Kowsher |
| collection | DOAJ |
| description | The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT,<xref ref-type="fn" rid="fn1">a</xref> a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%. |
| first_indexed | 2024-04-12T23:16:24Z |
| format | Article |
| id | doaj.art-97c72b1941a6454baf3dbd2436d7cf63 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T23:16:24Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-97c72b1941a6454baf3dbd2436d7cf632022-12-22T03:12:39ZengIEEEIEEE Access2169-35362022-01-0110918559187010.1109/ACCESS.2022.31976629852438Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language UnderstandingM. Kowsher0https://orcid.org/0000-0001-5469-3437Abdullah As Sami1https://orcid.org/0000-0002-5488-6973Nusrat Jahan Prottasha2Mohammad Shamsul Arefin3https://orcid.org/0000-0003-0259-7624Pranab Kumar Dhar4https://orcid.org/0000-0002-9664-3056Takeshi Koshiba5https://orcid.org/0000-0001-8994-729XDepartment of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USADepartment of Computer Science and Engineering, Chittagong University of Engineering and Technology, Chattogram, BangladeshDepartment of Computer Science and Engineering, Daffodil International University, Dhaka, BangladeshDepartment of Computer Science and Engineering, Daffodil International University, Dhaka, BangladeshChittagong University of Engineering and Technology, Chattogram, BangladeshWaseda University, Tokyo, Shinjuku-ku, JapanThe advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT,<xref ref-type="fn" rid="fn1">a</xref> a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.https://ieeexplore.ieee.org/document/9852438/Bangla NLPBERT-baselarge corpustransformer |
| spellingShingle | M. Kowsher Abdullah As Sami Nusrat Jahan Prottasha Mohammad Shamsul Arefin Pranab Kumar Dhar Takeshi Koshiba Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding IEEE Access Bangla NLP BERT-base large corpus transformer |
| title | Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding |
| title_full | Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding |
| title_fullStr | Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding |
| title_full_unstemmed | Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding |
| title_short | Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding |
| title_sort | bangla bert transformer based efficient model for transfer learning and language understanding |
| topic | Bangla NLP BERT-base large corpus transformer |
| url | https://ieeexplore.ieee.org/document/9852438/ |
| work_keys_str_mv | AT mkowsher banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding AT abdullahassami banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding AT nusratjahanprottasha banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding AT mohammadshamsularefin banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding AT pranabkumardhar banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding AT takeshikoshiba banglaberttransformerbasedefficientmodelfortransferlearningandlanguageunderstanding |