Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a dir...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2023-01-01
|
| Series: | ESAIM: Proceedings and Surveys |
| Online Access: | https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf |
| _version_ | 1797673457746444288 |
|---|---|
| author | Koroko Abdoulaye Anciaux-Sedrakian Ani Gharbia Ibtihel Ben Garès Valérie Haddou Mounir Tran Quang Huy |
| author_facet | Koroko Abdoulaye Anciaux-Sedrakian Ani Gharbia Ibtihel Ben Garès Valérie Haddou Mounir Tran Quang Huy |
| author_sort | Koroko Abdoulaye |
| collection | DOAJ |
| description | We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed. |
| first_indexed | 2024-03-11T21:45:42Z |
| format | Article |
| id | doaj.art-8a4fc72eae9e421f84c52e8da9df936d |
| institution | Directory Open Access Journal |
| issn | 2267-3059 |
| language | English |
| last_indexed | 2024-03-11T21:45:42Z |
| publishDate | 2023-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | ESAIM: Proceedings and Surveys |
| spelling | doaj.art-8a4fc72eae9e421f84c52e8da9df936d2023-09-26T10:13:00ZengEDP SciencesESAIM: Proceedings and Surveys2267-30592023-01-017321823710.1051/proc/202373218proc2307311Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decompositionKoroko Abdoulaye0Anciaux-Sedrakian Ani1Gharbia Ibtihel Ben2Garès Valérie3Haddou Mounir4Tran Quang Huy5IFP Energies nouvellesIFP Energies nouvellesIFP Energies nouvellesUniv Rennes, INSA, CNRS, IRMAR - UMR 6625Univ Rennes, INSA, CNRS, IRMAR - UMR 6625IFP Energies nouvellesWe design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf |
| spellingShingle | Koroko Abdoulaye Anciaux-Sedrakian Ani Gharbia Ibtihel Ben Garès Valérie Haddou Mounir Tran Quang Huy Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition ESAIM: Proceedings and Surveys |
| title | Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| title_full | Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| title_fullStr | Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| title_full_unstemmed | Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| title_short | Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| title_sort | efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition |
| url | https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf |
| work_keys_str_mv | AT korokoabdoulaye efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition AT anciauxsedrakianani efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition AT gharbiaibtihelben efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition AT garesvalerie efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition AT haddoumounir efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition AT tranquanghuy efficientapproximationsofthefishermatrixinneuralnetworksusingkroneckerproductsingularvaluedecomposition |