Most connectionist or "neural network" learning systems use some form of the back-propagation algorithm. However, back-propagation learning is too slow for many applications, and it scales up poorly as tasks become larger and more complex. The factors governing learning speed are poorly understood. I have begun a systematic, empirical study of learning speed in backprop-like algorithms, measured against a variety of benchmark problems. The goal is twofold: to develop faster learning algorithms and to contribute to the development of a methodology that will be of value in future studies of this kind. This paper is a progress report describing the results obtained during the first six months of this study. To date I have looked only at a limited set of benchmark problems, but the results on these are encouraging: I have developed a new learning algorithm that is faster than standard backprop by an order of magnitude or more and that appears to scale up very well as the problem size increases. This research was sponsored in part by the National Science Foundation under Contract Number EET-8716324 and by the Defense Advanced Research Projects Agency (DOD), ARPA Order No. 4976 under Contract F33615-87C-1499 and monitored by the Avionics Laboratory, Air Force Wright Aeronautical Laboratories, Aeronautical Systems Division (AFSC), Wright-Patterson AFB, OH 45433-6543. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of these agencies or of the U.S. Government.
[1]
Dean G. Blevins,et al.
Introduction 1-2
,
1969
.
[2]
P. Werbos,et al.
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
,
1974
.
[3]
Yann LeCun,et al.
Une procedure d'apprentissage pour reseau a seuil asymmetrique (A learning scheme for asymmetric threshold networks)
,
1985
.
[4]
Geoffrey E. Hinton,et al.
Learning internal representations by error propagation
,
1986
.
[5]
Geoffrey E. Hinton,et al.
Experiments on Learning by Back Propagation.
,
1986
.
[6]
James L. McClelland,et al.
Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations
,
1986
.
[7]
Bernardo A. Huberman,et al.
AN IMPROVED THREE LAYER, BACK PROPAGATION ALGORITHM
,
1987
.
[8]
Geoffrey E. Hinton,et al.
Connectionist Architectures for Artificial Intelligence
,
1990,
Computer.
[9]
Gerald Tesauro,et al.
Scaling Relationships in Back-propagation Learning
,
1988,
Complex Syst..
[10]
Raymond L. Watrous.
Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization
,
1988
.
[11]
D. S. Touretzky,et al.
Neural network simulation at Warp speed: how we got 17 million connections per second
,
1988,
IEEE 1988 International Conference on Neural Networks.
[12]
Robert A. Jacobs,et al.
Increased rates of convergence through learning rate adaptation
,
1987,
Neural Networks.
[13]
L. Steels,et al.
Accelerated Learning in Back-propagation Nets
,
1989
.