Optimization of the Backpropagation Algorithm for Training Multilayer Perceptrons

5 Global learning rate adaptation 8 5.1 Fixed calculating of the learning rate 8 5.2 Decreasing learning rate 8 5.3 Learning rate adaptation for each training pattern 12 5.4 Evolutionarily adapted learning rate 12 5.5 Angle driven learning rate adaptation 15 5.6 Nearly optimal learning rate adjust using line search 15 5.6.1 Polak–Ribiere method and line search 17 5.6.2 Conjugate gradient method and line search 18

[1]  E. Polak Introduction to linear and nonlinear programming , 1973 .

[2]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[3]  Frank Fallside,et al.  An adaptive training algorithm for back propagation networks , 1987 .

[4]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[5]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[6]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[7]  L. Steels,et al.  Accelerated Learning in Back-propagation Nets , 1989 .

[8]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[9]  Mark A. Kramer,et al.  Improvement of the backpropagation algorithm for training neural networks , 1990 .

[10]  Ralf Salomon Improved Convergence Rate of Back-Propagation with Dynamic Adaptation of the Learning Rate , 1990, PPSN.

[11]  John E. Moody,et al.  Note on Learning Rate Schedules for Stochastic Optimization , 1990, NIPS.

[12]  Luís B. Almeida,et al.  Speeding up Backpropagation , 1990 .

[13]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[14]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[15]  Frank Fallside,et al.  A recurrent error propagation network speech recognition system , 1991 .

[16]  Martin A. Riedmiller,et al.  RPROP - A Fast Adaptive Learning Algorithm , 1992 .

[17]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[18]  Harry A. C. Eaton,et al.  Learning coefficient dependence on training set size , 1992, Neural Networks.

[19]  Wolfram Schiffmann,et al.  Comparison of optimized backpropagation algorithms , 1993, ESANN.