Incremental Slow Feature Analysis: Adaptive Low-Complexity Slow Feature Updating from High-Dimensional Input Streams

We introduce here an incremental version of slow feature analysis (IncSFA), combining candid covariance-free incremental principal components analysis (CCIPCA) and covariance-free incremental minor components analysis (CIMCA). IncSFA's feature updating complexity is linear with respect to the input dimensionality, while batch SFA's (BSFA) updating complexity is cubic. IncSFA does not need to store, or even compute, any covariance matrices. The drawback to IncSFA is data efficiency: it does not use each data point as effectively as BSFA. But IncSFA allows SFA to be tractably applied, with just a few parameters, directly on high-dimensional input streams (e.g., visual input of an autonomous agent), while BSFA has to resort to hierarchical receptive-field-based architectures when the input dimension is too high. Further, IncSFA's updates have simple Hebbian and anti-Hebbian forms, extending the biological plausibility of SFA. Experimental results show IncSFA learns the same set of features as BSFA and can handle a few cases where BSFA fails.

[1]  J. Weng,et al.  Convergence Analysis of Complementary Candid Incremental Principal Component Analysis ∗ , 2001 .

[2]  Andrew McCallum,et al.  Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data , 2004, J. Mach. Learn. Res..

[3]  Vwani P. Roychowdhury,et al.  Algorithms for accelerated convergence of adaptive PCA , 2000, IEEE Trans. Neural Networks Learn. Syst..

[4]  S.-I. Amari,et al.  Neural theory of association and concept-formation , 1977, Biological Cybernetics.

[5]  Sridhar Mahadevan,et al.  Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes , 2007, J. Mach. Learn. Res..

[6]  Pierre Comon.  Independent component analysis - a new concept? signal processing , 1994 .

[7]  Shun-ichi Amari,et al.  Sequential Extraction of Minor Components , 2001, Neural Processing Letters.

[8]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[9]  Laurenz Wiskott.  Estimating Driving Forces of Nonstationary Time Series with Slow Feature Analysis Laurenz Wiskott Institute for Theoretical Biology , 2003 .

[10]  Jürgen Schmidhuber,et al.  Sequential Constant Size Compressors for Reinforcement Learning , 2011, AGI.

[11]  Charles W. Groetsch,et al.  Lanczos' Generalized Derivative , 1998 .

[12]  E. Rolls.  Spatial view cells and the representation of place in the primate hippocampus , 1999, Hippocampus.

[13]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[14]  G. Forsythe,et al.  The cyclic Jacobi method for computing the principal values of a complex matrix , 1960 .

[15]  R U Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[16]  T. Hafting,et al.  Microstructure of a spatial map in the entorhinal cortex , 2005, Nature.

[17]  Dezhong Peng,et al.  A New Algorithm for Sequential Minor Component Analysis , 2006 .

[18]  Erkki Oja,et al.  Modified Hebbian learning for curve and surface fitting , 1992, Neural Networks.

[19]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[20]  Bruno A. Olshausen,et al.  Book Review , 2003, Journal of Cognitive Neuroscience.

[21]  Laurenz Wiskott,et al.  Slowness and Sparseness Lead to Place, Head-Direction, and Spatial-View Cells , 2007, PLoS Comput. Biol..

[22]  Juha Karhunen,et al.  A Unified Neural Bigradient Algorithm for robust PCA and MCA , 1996, Int. J. Neural Syst..

[23]  Niko Wilbert,et al.  Modular Toolkit for Data Processing (MDP): A Python Data Processing Framework , 2008, Frontiers Neuroinformatics.

[24]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[25]  Qin Lin,et al.  A unified algorithm for principal and minor components extraction , 1998, Neural Networks.

[26]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[27]  Juyang Weng,et al.  Optimal In-Place Learning and the Lobe Component Analysis , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[28]  M. V. Velzen,et al.  Self-organizing maps , 2007 .

[29]  E. Rolls,et al.  INVARIANT FACE AND OBJECT RECOGNITION IN THE VISUAL SYSTEM , 1997, Progress in Neurobiology.

[30]  R. Muller,et al.  Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[31]  Terrence J. Sejnowski,et al.  Slow Feature Analysis: Unsupervised Learning of Invariances , 2002, Neural Computation.

[32]  Stefanie N. Lindstaedt,et al.  Comparison of two Unsupervised Neural Network Models for Redundancy Reduction , 1993 .

[33]  H Barlow,et al.  Redundancy reduction revisited , 2001, Network.

[34]  Jürgen Schmidhuber,et al.  Learning Complex, Extended Sequences Using the Principle of History Compression , 1992, Neural Computation.

[35]  Zhang Yi,et al.  Convergence analysis of a simple minor component analysis algorithm , 2007, Neural Networks.

[36]  Laurenz Wiskott,et al.  Slowness: An Objective for Spike-Timing–Dependent Plasticity? , 2007, PLoS Comput. Biol..

[37]  Robert A. Legenstein,et al.  Reinforcement Learning on Slow Features of High-Dimensional Input Streams , 2010, PLoS Comput. Biol..

[38]  Henning Sprekeler,et al.  On the Relation of Slow Feature Analysis and Laplacian Eigenmaps , 2011, Neural Computation.

[39]  Laurenz Wiskott,et al.  An extension of slow feature analysis for nonlinear blind source separation , 2014, J. Mach. Learn. Res..

[40]  I. Miller.  Probability, Random Variables, and Stochastic Processes , 1966 .

[41]  Graeme Mitchison,et al.  Removing Time Variation with the Anti-Hebbian Differential Synapse , 1991, Neural Computation.

[42]  Mark H. Johnson,et al.  Object Recognition and Sensitive Periods: A Computational Analysis of Visual Imprinting , 1994, Neural Computation.

[43]  Juyang Weng,et al.  Candid Covariance-Free Incremental Principal Component Analysis , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Peter Földiák,et al.  Learning Invariance from Transformation Sequences , 1991, Neural Comput..

[45]  Niko Wilbert,et al.  Slow feature analysis , 2011, Scholarpedia.

[46]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[47]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[48]  Honglak Lee,et al.  Unsupervised feature learning for audio classification using convolutional deep belief networks , 2009, NIPS.

[49]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[50]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[51]  Jürgen Schmidhuber,et al.  Learning Unambiguous Reduced Sequence Descriptions , 1991, NIPS.

[52]  R. Gray,et al.  Vector quantization , 1984, IEEE ASSP Magazine.

[53]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[54]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[55]  Yoshua Bengio,et al.  Slow, Decorrelated Features for Pretraining Complex Cell-like Networks , 2009, NIPS.

[56]  Jürgen Schmidhuber,et al.  Incremental Slow Feature Analysis , 2011, IJCAI.

[57]  Maja J. Mataric,et al.  A spatio-temporal extension to Isomap nonlinear dimension reduction , 2004, ICML.

[58]  S. Grossberg,et al.  How does a brain build a cognitive code? , 1980, Psychological review.

[59]  Angelo Cangelosi,et al.  An open-source simulator for cognitive robotics research: the prototype of the iCub humanoid robot simulator , 2008, PerMIS.

[60]  Jürgen Schmidhuber,et al.  AutoIncSFA and vision-based developmental learning for humanoid robots , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[61]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[62]  Jin Young Choi,et al.  Novel Incremental Principal Component Analysis with Improved Performance , 2008, SSPR/SPR.

[63]  J. Urgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992 .

[64]  Pratibha Mishra,et al.  Advanced Engineering Mathematics , 2013 .

[65]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[66]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[67]  Jürgen Schmidhuber,et al.  Unsupervised Learning in LSTM Recurrent Neural Networks , 2001, ICANN.

[68]  Jürgen Schmidhuber.  Neural Predictors for Detecting and Removing Redundant Information , 2000 .

[69]  Carl Doersch,et al.  Temporal Continuity Learning for Convolutional Deep Belief Networks , 2010 .