Theory of Monte Carlo sampling-based Alopex algorithms for neural networks

We propose two novel Monte Carlo sampling-based Alopex (ALgorithm Of Pattern EXtraction) algorithms for training neural networks. The proposed algorithms naturally combine the sequential Monte Carlo estimation and Alopex-like procedure for gradient-free optimization, and the learning proceeds within the recursive Bayesian estimation framework. Experimental results on various problems show encouraging convergence results.

[1]  Alejandro Bia.  Alopex-B: A New, Simpler, But Yet Faster Version Of The Alopex Training Algorithm , 2001, Int. J. Neural Syst..

[2]  W H Wong,et al.  Dynamic weighting in Monte Carlo and optimization. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Simon Haykin,et al.  Stochastic correlative learning algorithms , 2004, IEEE Transactions on Signal Processing.

[4]  Nando de Freitas,et al.  Sequential Monte Carlo Methods in Practice , 2001, Statistics for Engineering and Information Science.

[5]  E. Harth,et al.  The Alopex process: Visual receptive fields by response feedback , 1979, Biological Cybernetics.

[6]  Nando de Freitas,et al.  The Unscented Particle Filter , 2000, NIPS.

[7]  Evangelia Micheli-Tzanakou,et al.  Supervised and unsupervised pattern recognition: feature extraction and computational intelligence , 2000 .

[8]  E Harth,et al.  The inversion of sensory processing by feedback pathways: a model of visual cognitive functions. , 1987, Science.

[9]  P. Shanti Sastry,et al.  New algorithms for learning and pruning oblique decision trees , 1999, IEEE Trans. Syst. Man Cybern. Part C.

[10]  P. S. Sastry,et al.  Two Timescale Analysis of the Alopex Algorithm for Optimization , 2002, Neural Computation.

[11]  Volker Tresp,et al.  Fisher Scoring and a Mixture of Modes Approach for Approximate Inference and Learning in Nonlinear State Space Models , 1998, NIPS.

[12]  K. P. Unnikrishnan,et al.  Alopex: A Correlation-Based Learning Algorithm for Feedforward and Recurrent Neural Networks , 1994, Neural Computation.

[13]  David J. C. Mackay,et al.  Introduction to Monte Carlo Methods , 1998, Learning in Graphical Models.

[14]  Arnaud Doucet,et al.  Sequential Monte Carlo Methods to Train Neural Network Models , 2000, Neural Computation.