Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition

Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition

We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a dir...

Full description

Bibliographic Details
Main Authors: Koroko Abdoulaye, Anciaux-Sedrakian Ani, Gharbia Ibtihel Ben, Garès Valérie, Haddou Mounir, Tran Quang Huy
Format: Article
Language:English
Published: EDP Sciences 2023-01-01
Series:ESAIM: Proceedings and Surveys
Online Access:https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf

Internet

https://www.esaim-proc.org/articles/proc/pdf/2023/02/proc2307311.pdf

Similar Items