We calculate analytically the rate of convergence at long times in the backpropagation learning algorithm for networks with and without hidden units. For networks without hidden units using the standard quadratic error function and a sigmoidal transfer function, we find that the error decreases as 1/t for large t, and the output states approach their target values as 1/t. It is possible to obtain a different convergence rate for certain error and transfer functions, but the convergence can never be faster than 1/t. These results are unaffected by a momentum term in the learning algorithm, but convergence can be substantially improved by an adaptive learning rate scheme. For networks with hidden units, we generally expect the same rate of convergence to be obtained as in the single-layer case; however, under certain circumstances one can obtain a polynomial speed-up for non sigmoidal units, or a logarithmic speed-up for sigmoidal units. Our analytic results are confirmed by empirical measurements of the convergence rate in numerical simulations.
[1]
Geoffrey E. Hinton,et al.
Learning representations by back-propagating errors
,
1986,
Nature.
[2]
Bernard Widrow,et al.
Adaptive switching circuits
,
1988
.
[3]
Gerald Tesauro,et al.
A study of scaling and generalization in neural networks
,
1988,
Neural Networks.
[4]
Robert A. Jacobs,et al.
Increased rates of convergence through learning rate adaptation
,
1987,
Neural Networks.
[5]
Gerald Tesauro,et al.
Scaling and Generalization in Neural Networks: A Case Study
,
1988,
NIPS.
[6]
Geoffrey E. Hinton.
Connectionist Learning Procedures
,
1989,
Artif. Intell..
[7]
Richard S. Sutton,et al.
Learning a nonlinear model of a manufacturing process using multilayer connectionist networks
,
1990,
Proceedings. 5th IEEE International Symposium on Intelligent Control 1990.