Stochastic correlative learning algorithms

This paper addresses stochastic correlative learning as the basis for a broadly defined class of statistical learning algorithms known collectively as the algorithm of pattern extraction (ALOPEX) family. Starting with the neurobiologically motivated Hebb's rule, the two conventional forms of the ALOPEX algorithm are derived, followed by a modified variant designed to improve the convergence speed. We next describe two more elaborate versions of the ALOPEX algorithm, which incorporate particle filtering that exemplifies a form of Monte Carlo simulation, to exchange computational complexity for an improved convergence and tracking behavior. In support of the different forms of the ALOPEX algorithm developed herein, we present three different experiments using synthetic and real-life data on binocular fusion of stereo images, on-line prediction, and system identification.

[1]  W. James.  Psychology: Briefer Course , 2020 .

[2]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[3]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[4]  James A. Anderson,et al.  A simple neural network generating an interactive memory , 1972 .

[5]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[6]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[7]  J. Pokorny.  Foundations of Cyclopean Perception , 1972 .

[8]  F. Black,et al.  The Pricing of Options and Corporate Liabilities , 1973, Journal of Political Economy.

[9]  E Harth,et al.  Alopex: a stochastic method for determining visual receptive fields. , 1974, Vision research.

[10]  D Marr,et al.  Cooperative computation of stereo disparity. , 1976, Science.

[11]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[12]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[13]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[14]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  E Harth,et al.  The inversion of sensory processing by feedback pathways: a model of visual cognitive functions. , 1987, Science.

[16]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[17]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[18]  J. Eggermont.  The Correlative Brain: Theory and Experiment in Neural Interaction , 1990 .

[19]  James M. Hutchinson,et al.  A radial basis function approach to financial time series analysis , 1993 .

[20]  Osamu Fujita.  Trial-and-error correlation learning , 1993, IEEE Trans. Neural Networks.

[21]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[22]  Abhijit S. Pandya,et al.  A recurrent neural network controller and learning algorithm for the on-line learning control of autonomous underwater vehicles , 1994, Neural Networks.

[23]  K. P. Unnikrishnan,et al.  Alopex: A Correlation-Based Learning Algorithm for Feedforward and Recurrent Neural Networks , 1994, Neural Computation.

[24]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[25]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[26]  E. Micheli-Tzanakou,et al.  Neural networks in signal and image processing , 1996, Professional Program Proceedings. ELECTRO '96.

[27]  W H Wong,et al.  Dynamic weighting in Monte Carlo and optimization. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Volker Tresp,et al.  Fisher Scoring and a Mixture of Modes Approach for Approximate Inference and Learning in Nonlinear State Space Models , 1998, NIPS.

[29]  Nathan Intrator,et al.  Receptive Field Formation in Natural Scene Environments: Comparison of Single-Cell Learning Rules , 1997, Neural Computation.

[30]  P. Shanti Sastry,et al.  New algorithms for learning and pruning oblique decision trees , 1999, IEEE Trans. Syst. Man Cybern. Part C.

[31]  E. Micheli-Tzanakou,et al.  Neural networks trained with simulation data for outcome prediction in pallidotomy for Parkinson's disease , 2000, Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No.00CH37143).

[32]  Gomes de Freitas,et al.  Bayesian methods for neural networks , 2000 .

[33]  Nando de Freitas,et al.  The Unscented Particle Filter , 2000, NIPS.

[34]  Evangelia Micheli-Tzanakou,et al.  Supervised and unsupervised pattern recognition: feature extraction and computational intelligence , 2000 .

[35]  Arnaud Doucet,et al.  Sequential Monte Carlo Methods to Train Neural Network Models , 2000, Neural Computation.

[36]  Neil J. Gordon,et al.  Editors: Sequential Monte Carlo Methods in Practice , 2001 .

[37]  Alejandro Bia.  Alopex-B: A New, Simpler, But Yet Faster Version Of The Alopex Training Algorithm , 2001, Int. J. Neural Syst..

[38]  P. S. Sastry,et al.  Two Timescale Analysis of the Alopex Algorithm for Optimization , 2002, Neural Computation.

[39]  James T. Lo,et al.  Adaptive multilayer perceptrons with long- and short-term memories , 2002, IEEE Trans. Neural Networks.

[40]  Michael J. Anderson,et al.  Auditory stimulus optimization with feedback from fuzzy clustering of neuronal responses , 2002, IEEE Transactions on Information Technology in Biomedicine.

[41]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[42]  Simon Haykin,et al.  Theory of Monte Carlo sampling-based Alopex algorithms for neural networks , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[43]  E. Harth,et al.  The Alopex process: Visual receptive fields by response feedback , 1979, Biological Cybernetics.

[44]  J. A. Anderson,et al.  A memory storage model utilizing spatial correlation functions , 1968, Kybernetik.