Slow Feature Analysis: Unsupervised Learning of Invariances

Invariant features of temporally varying signals are useful for analysis and classification. Slow feature analysis (SFA) is a new method for learning invariant or slowly varying features from a vectorial input signal. It is based on a nonlinear expansion of the input signal and application of principal component analysis to this expanded signal and its time derivative. It is guaranteed to find the optimal solution within a family of functions directly and can learn to extract a large number of decor-related features, which are ordered by their degree of invariance. SFA can be applied hierarchically to process high-dimensional input signals and extract complex features. SFA is applied first to complex cell tuning properties based on simple cell output, including disparity and motion. Then more complicated input-output functions are learned by repeated application of SFA. Finally, a hierarchical network of SFA modules is presented as a simple model of the visual system. The same unstructured network can learn translation, size, rotation, contrast, or, to a lesser degree, illumination invariance for one-dimensional objects, depending on only the training stimulus. Surprisingly, only a few training objects suffice to achieve good generalization to new objects. The generated representation is suitable for object recognition. Performance degrades if the network is trained to learn multiple invariances simultaneously.

[1]  Takayuki Ito,et al.  Neocognitron: A neural network model for a mechanism of visual pattern recognition , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[2]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[3]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[4]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .

[5]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[6]  Geoffrey E. Hinton,et al.  Discovering Viewpoint-Invariant Relationships That Characterize Objects , 1990, NIPS.

[7]  Peter Földiák,et al.  Learning Invariance from Transformation Sequences , 1991, Neural Comput..

[8]  Graeme Mitchison,et al.  Removing Time Variation with the Anti-Hebbian Differential Synapse , 1991, Neural Computation.

[9]  Geoffrey E. Hinton,et al.  Self-organizing neural network that discovers surfaces in random-dot stereograms , 1992, Nature.

[10]  Suzanna Becker,et al.  Learning to Categorize Objects Using Temporal Coherence , 1992, NIPS.

[11]  Harry G. Barrow,et al.  A Model of Adaptive Development of Complex Cortical Cells , 1992 .

[12]  D. V. van Essen,et al.  A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[13]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[14]  Hanspeter A. Mallot,et al.  Phase-based binocular vergence control and depth reconstruction using active vision , 1994 .

[15]  Mark H. Johnson,et al.  Object Recognition and Sensitive Periods: A Computational Analysis of Visual Imprinting , 1994, Neural Computation.

[16]  Wolfgang Konen,et al.  A fast dynamic link matching algorithm for invariant pattern recognition , 1994, Neural Networks.

[17]  David I. Perrett,et al.  Modeling visual recognition from neurobiological constraints , 1994, Neural Networks.

[18]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[19]  James V. Stone,et al.  A learning rule for extracting spatio-temporal invariances , 1995 .

[20]  Geoffrey E. Hinton,et al.  Spatial coherence as an internal teacher for a neural network , 1995 .

[21]  Suzanna Becker,et al.  Mutual information maximization: models of cortical self-organization. , 1996, Network.

[22]  James V. Stone Learning Perceptually Salient Visual Parameters Using Spatiotemporal Smoothness Constraints , 1996, Neural Computation.

[23]  Michael Eisele Unsupervised learning of temporal constancies by pyramidal-type neurons , 1997 .

[24]  E. Rolls,et al.  INVARIANT FACE AND OBJECT RECOGNITION IN THE VISUAL SYSTEM , 1997, Progress in Neurobiology.

[25]  T J Sejnowski,et al.  Learning viewpoint-invariant face representations from visual experience in an attractor network. , 1998, Network.

[26]  Hanchuan Peng,et al.  Energy function for learning invariance in multilayer perceptron , 1998 .

[27]  Kunihiko Fukushima,et al.  Self-organization of shift-invariant receptive fields , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[28]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[29]  N. Krüger,et al.  Multi-modal estimation of collinearity and parallelism in natural image sequences. , 2002 .

[30]  A. Hyvärinen,et al.  Temporal and spatiotemporal coherence in simple-cell responses: a generative model of natural image sequences , 2003 .

[31]  Laurenz Wiskott,et al.  Is slowness a learning principle of the visual cortex? , 2003, Zoology.

[32]  Aapo Hyvärinen,et al.  Simple-Cell-Like Receptive Fields Maximize Temporal Coherence in Natural Video , 2003, Neural Computation.

[33]  Konrad P. Körding,et al.  Learning the Nonlinearity of Neurons from Natural Visual Stimuli , 2003, Neural Computation.

[34]  Laurenz Wiskott,et al.  Slow Feature Analysis: A Theoretical Analysis of Optimal Free Responses , 2003, Neural Computation.

[35]  Aapo Hyvärinen,et al.  Bubbles: a unifying framework for low-level statistical properties of natural image sequences. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[36]  W. Hashimoto Quadratic forms in natural images. , 2003 .

[37]  Klaus Obermayer,et al.  Modeling the adaptive visual system: a survey of principled approaches , 2003, Neural Networks.

[38]  Klaus Obermayer,et al.  Second-Order Statistics of Natural Images , 2002, Neurocomputing.

[39]  Heiko Wersing,et al.  Learning Optimized Features for Hierarchical Models of Invariant Object Recognition , 2003, Neural Computation.

[40]  P. König,et al.  Learning Distinct and Complementary Feature Selectivities from Natural Colour Videos , 2003, Reviews in the neurosciences.

[41]  David J. Fleet,et al.  Computation of component image velocity from local phase information , 1990, International Journal of Computer Vision.

[42]  Mark D. Plumbley,et al.  Unsupervised neural network learning procedures for feature extraction and classification , 1996, Applied Intelligence.