Optimizing Small BERTs Trained for German NER
Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP t...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-10-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/12/11/443 |
| _version_ | 1797509953604288512 |
|---|---|
| author | Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn |
| author_facet | Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn |
| author_sort | Jochen Zöllner |
| collection | DOAJ |
| description | Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants, such as ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: Class-Start-End tagging and a modified form of Linear Chain Conditional Random Fields. Furthermore, we introduce Whole-Word Attention, which reduces BERTs memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German Named Entity Recognition (NER) tasks, of which two are introduced by this article. |
| first_indexed | 2024-03-10T05:25:01Z |
| format | Article |
| id | doaj.art-c269d264e4014373a18286098ba93af0 |
| institution | Directory Open Access Journal |
| issn | 2078-2489 |
| language | English |
| last_indexed | 2024-03-10T05:25:01Z |
| publishDate | 2021-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj.art-c269d264e4014373a18286098ba93af02023-11-22T23:45:17ZengMDPI AGInformation2078-24892021-10-01121144310.3390/info12110443Optimizing Small BERTs Trained for German NERJochen Zöllner0Konrad Sperfeld1Christoph Wick2Roger Labahn3Institute of Mathematics, University of Rostock, 18057 Rostock, GermanyInstitute of Mathematics, University of Rostock, 18057 Rostock, GermanyPLANET AI GmbH Rostock, 18057 Rostock, GermanyInstitute of Mathematics, University of Rostock, 18057 Rostock, GermanyCurrently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants, such as ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: Class-Start-End tagging and a modified form of Linear Chain Conditional Random Fields. Furthermore, we introduce Whole-Word Attention, which reduces BERTs memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German Named Entity Recognition (NER) tasks, of which two are introduced by this article.https://www.mdpi.com/2078-2489/12/11/443named entity recognitionnatural language processingBERTGerman languagepre-trainingfine-tuning |
| spellingShingle | Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn Optimizing Small BERTs Trained for German NER Information named entity recognition natural language processing BERT German language pre-training fine-tuning |
| title | Optimizing Small BERTs Trained for German NER |
| title_full | Optimizing Small BERTs Trained for German NER |
| title_fullStr | Optimizing Small BERTs Trained for German NER |
| title_full_unstemmed | Optimizing Small BERTs Trained for German NER |
| title_short | Optimizing Small BERTs Trained for German NER |
| title_sort | optimizing small berts trained for german ner |
| topic | named entity recognition natural language processing BERT German language pre-training fine-tuning |
| url | https://www.mdpi.com/2078-2489/12/11/443 |
| work_keys_str_mv | AT jochenzollner optimizingsmallbertstrainedforgermanner AT konradsperfeld optimizingsmallbertstrainedforgermanner AT christophwick optimizingsmallbertstrainedforgermanner AT rogerlabahn optimizingsmallbertstrainedforgermanner |