Multi-Manifold Attention for Vision Transformers
Vision Transformers are very popular nowadays due to their state-of-the-art performance in several computer vision tasks, such as image classification and action recognition. Although their performance has been greatly enhanced through highly descriptive patch embeddings and hierarchical structures,...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10305583/ |
| _version_ | 1797632774872498176 |
|---|---|
| author | Dimitrios Konstantinidis Ilias Papastratis Kosmas Dimitropoulos Petros Daras |
| author_facet | Dimitrios Konstantinidis Ilias Papastratis Kosmas Dimitropoulos Petros Daras |
| author_sort | Dimitrios Konstantinidis |
| collection | DOAJ |
| description | Vision Transformers are very popular nowadays due to their state-of-the-art performance in several computer vision tasks, such as image classification and action recognition. Although their performance has been greatly enhanced through highly descriptive patch embeddings and hierarchical structures, there is still limited research on utilizing additional data representations so as to refine the self-attention map of a Transformer. To address this problem, a novel attention mechanism, called multi-manifold multi-head attention, is proposed in this work to substitute the vanilla self-attention of a Transformer. The proposed mechanism models the input space in three distinct manifolds, namely Euclidean, Symmetric Positive Definite and Grassmann, thus leveraging different statistical and geometrical properties of the input for the computation of a highly descriptive attention map. In this way, the proposed attention mechanism can guide a Vision Transformer to become more attentive towards important appearance, color and texture features of an image, leading to improved classification and segmentation results, as shown by the experimental results on well-known datasets. |
| first_indexed | 2024-03-11T11:42:34Z |
| format | Article |
| id | doaj.art-7be00d85229b4931bc0886271ab5ac48 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T11:42:34Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-7be00d85229b4931bc0886271ab5ac482023-11-10T00:00:51ZengIEEEIEEE Access2169-35362023-01-011112343312344410.1109/ACCESS.2023.332995210305583Multi-Manifold Attention for Vision TransformersDimitrios Konstantinidis0https://orcid.org/0000-0002-7391-6875Ilias Papastratis1https://orcid.org/0000-0003-4664-2626Kosmas Dimitropoulos2https://orcid.org/0000-0003-1584-7047Petros Daras3https://orcid.org/0000-0003-3814-6710Information Technologies Institute, Centre for Research and Technology Hellas (CERTH), Thessaloniki, GreeceInformation Technologies Institute, Centre for Research and Technology Hellas (CERTH), Thessaloniki, GreeceInformation Technologies Institute, Centre for Research and Technology Hellas (CERTH), Thessaloniki, GreeceInformation Technologies Institute, Centre for Research and Technology Hellas (CERTH), Thessaloniki, GreeceVision Transformers are very popular nowadays due to their state-of-the-art performance in several computer vision tasks, such as image classification and action recognition. Although their performance has been greatly enhanced through highly descriptive patch embeddings and hierarchical structures, there is still limited research on utilizing additional data representations so as to refine the self-attention map of a Transformer. To address this problem, a novel attention mechanism, called multi-manifold multi-head attention, is proposed in this work to substitute the vanilla self-attention of a Transformer. The proposed mechanism models the input space in three distinct manifolds, namely Euclidean, Symmetric Positive Definite and Grassmann, thus leveraging different statistical and geometrical properties of the input for the computation of a highly descriptive attention map. In this way, the proposed attention mechanism can guide a Vision Transformer to become more attentive towards important appearance, color and texture features of an image, leading to improved classification and segmentation results, as shown by the experimental results on well-known datasets.https://ieeexplore.ieee.org/document/10305583/Attentionmanifoldvision transformersimage classificationsemantic segmentation |
| spellingShingle | Dimitrios Konstantinidis Ilias Papastratis Kosmas Dimitropoulos Petros Daras Multi-Manifold Attention for Vision Transformers IEEE Access Attention manifold vision transformers image classification semantic segmentation |
| title | Multi-Manifold Attention for Vision Transformers |
| title_full | Multi-Manifold Attention for Vision Transformers |
| title_fullStr | Multi-Manifold Attention for Vision Transformers |
| title_full_unstemmed | Multi-Manifold Attention for Vision Transformers |
| title_short | Multi-Manifold Attention for Vision Transformers |
| title_sort | multi manifold attention for vision transformers |
| topic | Attention manifold vision transformers image classification semantic segmentation |
| url | https://ieeexplore.ieee.org/document/10305583/ |
| work_keys_str_mv | AT dimitrioskonstantinidis multimanifoldattentionforvisiontransformers AT iliaspapastratis multimanifoldattentionforvisiontransformers AT kosmasdimitropoulos multimanifoldattentionforvisiontransformers AT petrosdaras multimanifoldattentionforvisiontransformers |