Anti-Hebbian learning in topologically constrained linear networks: a tutorial

Using standard results from the adaptive signal processing literature, we review the learning behavior of various constrained linear neural networks made up of anti-Hebbian synapses, where learning is driven by the criterion of minimizing the node information energy. We point out how simple learning rules of Hebbian type can provide fast self-organization, under rather wide connectivity constraints. We verify the results of the theory in a set of simulations.

[1]  H. Rutishauser.  Computational aspects of F. L. Bauer's simultaneous iteration method , 1969 .

[2]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[3]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  Gene H. Golub,et al.  Matrix computations , 1983 .

[5]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[6]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[7]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[8]  A. Harry Klopf,et al.  A drive-reinforcement model of single neuron function , 1987 .

[9]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[10]  Gilbert Strang,et al.  Introduction to applied mathematics , 1988 .

[11]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[12]  B. Kosco.  Differential Hebbian learning , 1987 .

[13]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[14]  S. Grossberg.  Neural Networks and Natural Intelligence , 1988 .

[15]  Sophocles J. Orfanidis,et al.  Optimum signal processing: An introduction. Second Edition , 1988 .

[16]  Sophocles J. Orfanidis,et al.  Optimum Signal Processing: An Introduction , 1988 .

[17]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[18]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[19]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[20]  Peter Földiák,et al.  Adaptation and decorrelation in the cortex , 1989 .

[21]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[22]  Ralph Linsker,et al.  How to Generate Ordered Maps by Maximizing the Mutual Information between Input and Output Signals , 1989, Neural Computation.

[23]  Sun-Yuan Kung,et al.  A neural network learning algorithm for adaptive principal component extraction (APEX) , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[24]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[25]  Sophocles J. Orfanidis,et al.  GramSchmidt Neural Nets , 1990, Neural Computation.

[26]  Peter Földiák,et al.  Learning Invariance from Transformation Sequences , 1991, Neural Comput..

[27]  Graeme Mitchison,et al.  Removing Time Variation with the Anti-Hebbian Differential Synapse , 1991, Neural Computation.

[28]  F. Palmieri,et al.  A comparison of two eigen-networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[29]  R. Vaccaro.  SVD and Signal Processing II: Algorithms, Analysis and Applications , 1991 .

[30]  Francesco Palmieri,et al.  Linear neural networks which minimize the output variance , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[31]  Chi-Tsong Chen,et al.  Linear System Theory and Design , 1995 .