A Lite Romanian BERT: ALR-BERT
Large-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model’s size in order to outperform the best prev...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Computers |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-431X/11/4/57 |
| _version_ | 1797436287459786752 |
|---|---|
| author | Dragoş Constantin Nicolae Rohan Kumar Yadav Dan Tufiş |
| author_facet | Dragoş Constantin Nicolae Rohan Kumar Yadav Dan Tufiş |
| author_sort | Dragoş Constantin Nicolae |
| collection | DOAJ |
| description | Large-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model’s size in order to outperform the best previously obtained performances. However, at some point, increasing the model’s parameters may lead to reaching its saturation point due to the limited capacity of GPU/TPU. In addition to this, such models are mostly available in English or a shared multilingual structure. Hence, in this paper, we propose a lite BERT trained on a large corpus solely in the Romanian language, which we called “A Lite Romanian BERT (ALR-BERT)”. Based on comprehensive empirical results, ALR-BERT produces models that scale far better than the original Romanian BERT. Alongside presenting the performance on downstream tasks, we detail the analysis of the training process and its parameters. We also intend to distribute our code and model as an open source together with the downstream task. |
| first_indexed | 2024-03-09T10:59:24Z |
| format | Article |
| id | doaj.art-ad4b78fb3c1b43369d7205561c284964 |
| institution | Directory Open Access Journal |
| issn | 2073-431X |
| language | English |
| last_indexed | 2024-03-09T10:59:24Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Computers |
| spelling | doaj.art-ad4b78fb3c1b43369d7205561c2849642023-12-01T01:22:30ZengMDPI AGComputers2073-431X2022-04-011145710.3390/computers11040057A Lite Romanian BERT: ALR-BERTDragoş Constantin Nicolae0Rohan Kumar Yadav1Dan Tufiş2Research Institute for Artificial Intelligence, Romanian Academy, 050711 Bucharest, RomaniaDepartment of Information and Communication, University of Agder, 4604 Grimstad, NorwayResearch Institute for Artificial Intelligence, Romanian Academy, 050711 Bucharest, RomaniaLarge-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model’s size in order to outperform the best previously obtained performances. However, at some point, increasing the model’s parameters may lead to reaching its saturation point due to the limited capacity of GPU/TPU. In addition to this, such models are mostly available in English or a shared multilingual structure. Hence, in this paper, we propose a lite BERT trained on a large corpus solely in the Romanian language, which we called “A Lite Romanian BERT (ALR-BERT)”. Based on comprehensive empirical results, ALR-BERT produces models that scale far better than the original Romanian BERT. Alongside presenting the performance on downstream tasks, we detail the analysis of the training process and its parameters. We also intend to distribute our code and model as an open source together with the downstream task.https://www.mdpi.com/2073-431X/11/4/57BERTtransformersALBERTNLPRomanian |
| spellingShingle | Dragoş Constantin Nicolae Rohan Kumar Yadav Dan Tufiş A Lite Romanian BERT: ALR-BERT Computers BERT transformers ALBERT NLP Romanian |
| title | A Lite Romanian BERT: ALR-BERT |
| title_full | A Lite Romanian BERT: ALR-BERT |
| title_fullStr | A Lite Romanian BERT: ALR-BERT |
| title_full_unstemmed | A Lite Romanian BERT: ALR-BERT |
| title_short | A Lite Romanian BERT: ALR-BERT |
| title_sort | lite romanian bert alr bert |
| topic | BERT transformers ALBERT NLP Romanian |
| url | https://www.mdpi.com/2073-431X/11/4/57 |
| work_keys_str_mv | AT dragosconstantinnicolae aliteromanianbertalrbert AT rohankumaryadav aliteromanianbertalrbert AT dantufis aliteromanianbertalrbert AT dragosconstantinnicolae literomanianbertalrbert AT rohankumaryadav literomanianbertalrbert AT dantufis literomanianbertalrbert |