A survey of transformers
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KeAi Communications Co. Ltd.
2022-01-01
|
| Series: | AI Open |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666651022000146 |
| _version_ | 1828087385067356160 |
|---|---|
| author | Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu |
| author_facet | Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu |
| author_sort | Tianyang Lin |
| collection | DOAJ |
| description | Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research. |
| first_indexed | 2024-04-11T05:11:35Z |
| format | Article |
| id | doaj.art-26c69b88a20e47fb8f18f209ce250005 |
| institution | Directory Open Access Journal |
| issn | 2666-6510 |
| language | English |
| last_indexed | 2024-04-11T05:11:35Z |
| publishDate | 2022-01-01 |
| publisher | KeAi Communications Co. Ltd. |
| record_format | Article |
| series | AI Open |
| spelling | doaj.art-26c69b88a20e47fb8f18f209ce2500052022-12-25T04:19:31ZengKeAi Communications Co. Ltd.AI Open2666-65102022-01-013111132A survey of transformersTianyang Lin0Yuxin Wang1Xiangyang Liu2Xipeng Qiu3School of Computer Science, Fudan University, Shanghai, 200433, China; Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, 200433, ChinaSchool of Computer Science, Fudan University, Shanghai, 200433, China; Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, 200433, ChinaSchool of Computer Science, Fudan University, Shanghai, 200433, China; Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, 200433, ChinaCorresponding author at: School of Computer Science, Fudan University, Shanghai, 200433, China; School of Computer Science, Fudan University, Shanghai, 200433, China; Shanghai Key Laboratory of Intelligent Information Processing, Shanghai, 200433, ChinaTransformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.http://www.sciencedirect.com/science/article/pii/S2666651022000146TransformerSelf-attentionPre-trained modelsDeep learning |
| spellingShingle | Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu A survey of transformers AI Open Transformer Self-attention Pre-trained models Deep learning |
| title | A survey of transformers |
| title_full | A survey of transformers |
| title_fullStr | A survey of transformers |
| title_full_unstemmed | A survey of transformers |
| title_short | A survey of transformers |
| title_sort | survey of transformers |
| topic | Transformer Self-attention Pre-trained models Deep learning |
| url | http://www.sciencedirect.com/science/article/pii/S2666651022000146 |
| work_keys_str_mv | AT tianyanglin asurveyoftransformers AT yuxinwang asurveyoftransformers AT xiangyangliu asurveyoftransformers AT xipengqiu asurveyoftransformers AT tianyanglin surveyoftransformers AT yuxinwang surveyoftransformers AT xiangyangliu surveyoftransformers AT xipengqiu surveyoftransformers |