Data Symmetries and Learning in Fully Connected Neural Networks
Symmetries in the data and how they constrain the learned weights of modern deep networks is still an open problem. In this work we study the simple case of fully connected shallow non-linear neural networks and consider two types of symmetries: full dataset symmetries where the dataset <inline-f...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10122571/ |
| _version_ | 1797823427182067712 |
|---|---|
| author | Fabio Anselmi Luca Manzoni Alberto D'onofrio Alex Rodriguez Giulio Caravagna Luca Bortolussi Francesca Cairoli |
| author_facet | Fabio Anselmi Luca Manzoni Alberto D'onofrio Alex Rodriguez Giulio Caravagna Luca Bortolussi Francesca Cairoli |
| author_sort | Fabio Anselmi |
| collection | DOAJ |
| description | Symmetries in the data and how they constrain the learned weights of modern deep networks is still an open problem. In this work we study the simple case of fully connected shallow non-linear neural networks and consider two types of symmetries: full dataset symmetries where the dataset <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula> is mapped into itself by any transformation <inline-formula> <tex-math notation="LaTeX">$g$ </tex-math></inline-formula>, i.e. <inline-formula> <tex-math notation="LaTeX">$gX=X$ </tex-math></inline-formula> or single data point symmetries where <inline-formula> <tex-math notation="LaTeX">$gx=x$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$x\in X$ </tex-math></inline-formula>. We prove and experimentally confirm that symmetries in the data are directly inherited at the level of the network’s learned weights and relate these findings with the common practice of data augmentation in modern machine learning. Finally, we show how symmetry constraints have a profound impact on the spectrum of the learned weights, an aspect of the so-called network implicit bias. |
| first_indexed | 2024-03-13T10:23:51Z |
| format | Article |
| id | doaj.art-6ef8851554c34f7c87ec8206b6585ca6 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-13T10:23:51Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-6ef8851554c34f7c87ec8206b6585ca62023-05-19T23:01:11ZengIEEEIEEE Access2169-35362023-01-0111472824729010.1109/ACCESS.2023.327493810122571Data Symmetries and Learning in Fully Connected Neural NetworksFabio Anselmi0https://orcid.org/0000-0002-0264-4761Luca Manzoni1https://orcid.org/0000-0001-6312-7728Alberto D'onofrio2Alex Rodriguez3Giulio Caravagna4https://orcid.org/0000-0003-4240-3265Luca Bortolussi5https://orcid.org/0000-0001-8874-4001Francesca Cairoli6Department of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalyDepartment of Mathematics and Geosciences, University of Trieste, Trieste, ItalySymmetries in the data and how they constrain the learned weights of modern deep networks is still an open problem. In this work we study the simple case of fully connected shallow non-linear neural networks and consider two types of symmetries: full dataset symmetries where the dataset <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula> is mapped into itself by any transformation <inline-formula> <tex-math notation="LaTeX">$g$ </tex-math></inline-formula>, i.e. <inline-formula> <tex-math notation="LaTeX">$gX=X$ </tex-math></inline-formula> or single data point symmetries where <inline-formula> <tex-math notation="LaTeX">$gx=x$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$x\in X$ </tex-math></inline-formula>. We prove and experimentally confirm that symmetries in the data are directly inherited at the level of the network’s learned weights and relate these findings with the common practice of data augmentation in modern machine learning. Finally, we show how symmetry constraints have a profound impact on the spectrum of the learned weights, an aspect of the so-called network implicit bias.https://ieeexplore.ieee.org/document/10122571/Artificial neural networkssymmetry invarianceequivariance |
| spellingShingle | Fabio Anselmi Luca Manzoni Alberto D'onofrio Alex Rodriguez Giulio Caravagna Luca Bortolussi Francesca Cairoli Data Symmetries and Learning in Fully Connected Neural Networks IEEE Access Artificial neural networks symmetry invariance equivariance |
| title | Data Symmetries and Learning in Fully Connected Neural Networks |
| title_full | Data Symmetries and Learning in Fully Connected Neural Networks |
| title_fullStr | Data Symmetries and Learning in Fully Connected Neural Networks |
| title_full_unstemmed | Data Symmetries and Learning in Fully Connected Neural Networks |
| title_short | Data Symmetries and Learning in Fully Connected Neural Networks |
| title_sort | data symmetries and learning in fully connected neural networks |
| topic | Artificial neural networks symmetry invariance equivariance |
| url | https://ieeexplore.ieee.org/document/10122571/ |
| work_keys_str_mv | AT fabioanselmi datasymmetriesandlearninginfullyconnectedneuralnetworks AT lucamanzoni datasymmetriesandlearninginfullyconnectedneuralnetworks AT albertodonofrio datasymmetriesandlearninginfullyconnectedneuralnetworks AT alexrodriguez datasymmetriesandlearninginfullyconnectedneuralnetworks AT giuliocaravagna datasymmetriesandlearninginfullyconnectedneuralnetworks AT lucabortolussi datasymmetriesandlearninginfullyconnectedneuralnetworks AT francescacairoli datasymmetriesandlearninginfullyconnectedneuralnetworks |