Alopex: A Correlation-Based Learning Algorithm for Feedforward and Recurrent Neural Networks

We present a learning algorithm for neural networks, called Alopex. Instead of error gradient, Alopex uses local correlations between changes in individual weights and changes in the global error measure. The algorithm does not make any assumptions about transfer functions of individual neurons, and does not explicitly depend on the functional form of the error measure. Hence, it can be used in networks with arbitrary transfer functions and for minimizing a large class of error measures. The learning algorithm is the same for feedforward and recurrent networks. All the weights in a network are updated simultaneously, using only local computations. This allows complete parallelization of the algorithm. The algorithm is stochastic and it uses a temperature parameter in a manner similar to that in simulated annealing. A heuristic annealing schedule is presented that is effective in finding global minima of error surfaces. In this paper, we report extensive simulation studies illustrating these advantages and show that learning times are comparable to those for standard gradient descent methods. Feedforward networks trained with Alopex are used to solve the MONK's problems and symmetry problems. Recurrent networks trained with the same algorithm are used for solving temporal XOR problems. Scaling properties of the algorithm are demonstrated using encoder problems of different sizes and advantages of appropriate error measures are illustrated using a variety of problems.

[1]  Y. T. Li,et al.  Principles of optimalizing control systems and an application to the internal combusion engine , 1951 .

[2]  J. Orbach Principles of Neurodynamics. Perceptrons and the Theory of Brain Mechanisms. , 1962 .

[3]  E Harth,et al.  Alopex: a stochastic method for determining visual receptive fields. , 1974, Vision Research.

[4]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[5]  E. Harth,et al.  Brainstem control of sensory information: a mechanism for perception. , 1985, International Journal of Psychophysiology.

[6]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[7]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[8]  Geoffrey E. Hinton,et al.  Learning symmetry groups with hidden units: beyond the perceptron , 1986 .

[9]  Matthew Clinton Michaelis Learning in a hierarchical neural network , 1986 .

[10]  Andrew G. Barto,et al.  Game-theoretic cooperativity in networks of self-interested units , 1987 .

[11]  E Harth,et al.  The inversion of sensory processing by feedback pathways: a model of visual cognitive functions. , 1987, Science.

[12]  Eric B. Baum,et al.  Supervised Learning of Probability Distributions by Neural Networks , 1987, NIPS.

[13]  J. J. Hopfield,et al.  Learning algorithms andprobability distributions infeed-forward andfeed-back networks , 1987 .

[14]  Snehasis Mukhopadhyay,et al.  Associative learning of Boolean functions , 1989, IEEE Trans. Syst. Man Cybern..

[15]  Kumpati S. Narendra,et al.  Learning automata - an introduction , 1989 .

[16]  Carsten Peterson,et al.  Explorations of the mean field theory learning algorithm , 1989, Neural Networks.

[17]  Geoffrey E. Hinton Connectionist Learning Procedures , 1989, Artif. Intell..

[18]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[19]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[20]  D. Ts'o,et al.  Functional organization of primate visual cortex revealed by high resolution optical imaging. , 1990, Science.

[21]  E. W. Kairiss,et al.  Hebbian synapses: biophysical mechanisms and algorithms. , 1990, Annual review of neuroscience.

[22]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990 .

[23]  Thomas Kailath,et al.  Model-free distributed learning , 1990, IEEE Trans. Neural Networks.

[24]  A. Antonini,et al.  Pioneer neurons and target selection in cerebral cortical development. , 1990, Cold Spring Harbor symposia on quantitative biology.

[25]  Abhijit S. Pandya,et al.  Continuous recognition of sonar targets using neural networks , 1991, Defense, Security, and Sensing.

[26]  John J. Hopfield,et al.  Connected-digit speaker-dependent speech recognition using a neural network with time-delayed connections , 1991, IEEE Trans. Signal Process..

[27]  Marwan A. Jabri,et al.  Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks , 1991, Neural Comput..

[28]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[29]  K. P. Unnikrishnan,et al.  Learning in connectionist networks using the Alopex algorithm , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[30]  Ron Meir,et al.  A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks , 1992, NIPS.

[31]  K. P. Unnikrishnan,et al.  A feedback model of visual attention , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[32]  Abhijit S. Pandya,et al.  Invariant recognition of 2-D objects using Alopex neural networks , 1992, Defense, Security, and Sensing.

[33]  K. P. Unnikrishnan,et al.  Dynamical Control of Visual Attention Through Feedback Pathways: A Network Model , 1993 .

[34]  H. S. Nine,et al.  The Role of Subplate Feedback in the Development of Ocular Dominance Columns , 1993 .

[35]  Abhijit S. Pandya,et al.  A recurrent neural network controller and learning algorithm for the on-line learning control of autonomous underwater vehicles , 1994, Neural Networks.

[36]  E. Micheli-Tzanakou,et al.  Alopex neural networks for manual alphabet recognition , 1994, Proceedings of 16th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Evor L. Hines,et al.  Integer-weight neural nets , 1994 .

[38]  P. S. Sastry,et al.  Memory neuron networks for identification and control of dynamical systems , 1994, IEEE Trans. Neural Networks.