论文信息 - Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons

Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons

The natural gradient descent method is applied to train an n-m-1 multilayer perceptron. Based on an efficient scheme to represent the Fisher information matrix for an n-m-1 stochastic multilayer perceptron, a new algorithm is proposed to calculate the natural gradient without inverting the Fisher information matrix explicitly. When the input dimension n is much larger than the number of hidden neurons m, the time complexity of computing the natural gradient is O(n).

Shun-ichi Amari | Howard Hua Yang | S. Amari | H. Yang

[1] G. Stewart. Introduction to matrix computations , 1973 .

[2] Saad,et al. On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[3] Todd K. Leen,et al. Using Curvature Information for Fast Stochastic Search , 1996, NIPS.

[4] Shun-ichi Amari,et al. Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient , 1996, NIPS.

[5] Shun-ichi Amari,et al. The Efficiency and the Robustness of Natural Gradient Descent Learning Rule , 1997, NIPS.

[6] Howard Hua Yang,et al. Natural Gradient Descent for Training Multi-Layer Perceptrons , 1997 .

[7] S. Amari,et al. Training Multi-Layer Perceptrons by Natural Gradient Descent , 1997, ICONIP.

[8] Shun-ichi Amari,et al. Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[9] S. Amari,et al. Statistical inference: learning in artificial neural networks , 1998, Trends in Cognitive Sciences.

[10] M. Rattray,et al. Analysis of natural gradient descent for multilayer neural networks , 1999, cond-mat/9901212.

[11] S. Amari. Natural Gradient Works Eciently in Learning , 2022 .