Statistical Mechanical Analysis of Online Learning with Weight Normalization in Single Layer Perceptron

Weight normalization, a newly proposed optimization method for neural networks by Salimans and Kingma (2016), decomposes the weight vector of a neural network into a radial length and a direction v...

[1]  Michael Biehl,et al.  On-line backpropagation in two-layered neural networks , 1995 .

[2]  Saad,et al.  On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[3]  Michael Biehl,et al.  Learning by on-line gradient descent , 1995 .

[4]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[5]  Hyeyoung Park,et al.  Slow Dynamics Due to Singularities of Hierarchical Learning Machines , 2005 .

[6]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[7]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.