Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a dir...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2023-01-01
|
| Series: | ESAIM: Proceedings and Surveys |
| Online Access: | https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf |