From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
With the recent advances in deep learning, different approaches to improving pre-trained language models (PLMs) have been proposed. PLMs have advanced state-of-the-art (SOTA) performance on various natural language processing (NLP) tasks such as machine translation, text classification, question ans...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-09-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/17/8805 |
| _version_ | 1797496260732649472 |
|---|---|
| author | Mourad Mars |
| author_facet | Mourad Mars |
| author_sort | Mourad Mars |
| collection | DOAJ |
| description | With the recent advances in deep learning, different approaches to improving pre-trained language models (PLMs) have been proposed. PLMs have advanced state-of-the-art (SOTA) performance on various natural language processing (NLP) tasks such as machine translation, text classification, question answering, text summarization, information retrieval, recommendation systems, named entity recognition, etc. In this paper, we provide a comprehensive review of prior embedding models as well as current breakthroughs in the field of PLMs. Then, we analyse and contrast the various models and provide an analysis of the way they have been built (number of parameters, compression techniques, etc.). Finally, we discuss the major issues and future directions for each of the main points. |
| first_indexed | 2024-03-10T03:01:07Z |
| format | Article |
| id | doaj.art-bfe5448ccfc141ecb2e01a278b3f1985 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T03:01:07Z |
| publishDate | 2022-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-bfe5448ccfc141ecb2e01a278b3f19852023-11-23T12:47:09ZengMDPI AGApplied Sciences2076-34172022-09-011217880510.3390/app12178805From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art WalkthroughMourad Mars0College of Computers and Information Systems, Umm Al-Qura University, Mecca 24382, Saudi ArabiaWith the recent advances in deep learning, different approaches to improving pre-trained language models (PLMs) have been proposed. PLMs have advanced state-of-the-art (SOTA) performance on various natural language processing (NLP) tasks such as machine translation, text classification, question answering, text summarization, information retrieval, recommendation systems, named entity recognition, etc. In this paper, we provide a comprehensive review of prior embedding models as well as current breakthroughs in the field of PLMs. Then, we analyse and contrast the various models and provide an analysis of the way they have been built (number of parameters, compression techniques, etc.). Finally, we discuss the major issues and future directions for each of the main points.https://www.mdpi.com/2076-3417/12/17/8805artificial intelligenceNLPpre-trained language model |
| spellingShingle | Mourad Mars From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough Applied Sciences artificial intelligence NLP pre-trained language model |
| title | From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough |
| title_full | From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough |
| title_fullStr | From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough |
| title_full_unstemmed | From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough |
| title_short | From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough |
| title_sort | from word embeddings to pre trained language models a state of the art walkthrough |
| topic | artificial intelligence NLP pre-trained language model |
| url | https://www.mdpi.com/2076-3417/12/17/8805 |
| work_keys_str_mv | AT mouradmars fromwordembeddingstopretrainedlanguagemodelsastateoftheartwalkthrough |