Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition - Koroko Abdoulaye; Anciaux-Sedrakian Ani; Gharbia Ibtihel Ben; Garès Valérie; Haddou Mounir; Tran Quang Huy (ESAIM: Proceedings and Surveys, 2023)
We design four novel approximations of the Fisher Information Matrix (FIM) that plays a central role in natural gradient descent methods for neural networks. The newly proposed approximations are aimed at improving Martens and Grosse’s Kronecker-factored block diagonal (KFAC) one. They rely on a direct minimization problem, the solution of which can be computed via the Kronecker product singular value decomposition technique. Experimental results on the three standard deep auto-encoder benchmarks showed that they provide more accurate approximations to the FIM. Furthermore, they outperform KFAC and state-of-the-art first-order methods in terms of optimization speed.