Low-Complexity Coding and Decoding

We present a novel approach to sensory coding and unsu-pervised learning. It is called \Low-complexity coding and decoding" (Lococode). Unlike previous methods it explicitly takes into account the information-theoretic complexity of the code generator: lococodes (1) convey information about the input data and (2) can be computed and decoded by low-complexity mappings. To implement Lococode we train autoassociators with Flat Minimum Search, a recent method for discovering neural nets that can be described with few bits of information. Experiments show: unlike codes obtained with standard autoencoders, lococodes are based on familiar feature detectors, never unstructured, usually sparse, sometimes factorial or local (depending on the data). Unlike, e.g., independent component analysis (ICA) Lococode does not need to know the number of independent data sources.

[1]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[2]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[3]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[4]  Terrence J. Sejnowski,et al.  Unsupervised Discrimination of Clustered Data via Optimization of Binary Information Gain , 1992, NIPS.

[5]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[6]  Suzanna Becker,et al.  Unsupervised Learning Procedures for Neural Networks , 1991, Int. J. Neural Syst..

[7]  Geoffrey E. Hinton,et al.  Developing Population Codes by Minimizing Description Length , 1993, NIPS.

[8]  Jürgen Schmidhuber,et al.  Unsupervised Coding with LOCOCODE , 1997, ICANN.

[9]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[10]  Jürgen Schmidhuber,et al.  Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability , 1997, Neural Networks.

[11]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[12]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[13]  Peter Földiák,et al.  Sparse coding in the primate cortex , 1998 .

[14]  Zhaoping Li,et al.  A Theory of the Visual Motion Coding in the Primary Visual Cortex , 1996, Neural Computation.

[15]  J. Urgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992 .

[16]  R. Zemel,et al.  Learning sparse multiple cause models , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[17]  Jürgen Schmidhuber,et al.  Semilinear Predictability Minimization Produces Well-Known Feature Detectors , 1996, Neural Computation.

[18]  Jürgen Schmidhuber,et al.  Low-Complexity Art , 2017 .

[19]  Satosi Watanabe,et al.  Pattern Recognition: Human and Mechanical , 1985 .

[20]  Geoffrey E. Hinton,et al.  Generative models for discovering sparse distributed representations. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[21]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[22]  M. Kramer Nonlinear principal component analysis using autoassociative neural networks , 1991 .

[23]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[24]  Günther Palm,et al.  On the Information Storage Capacity of Local Learning Rules , 1992, Neural Computation.

[25]  David Zipser,et al.  Feature discovery by competitive learning , 1986 .

[26]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[27]  Garrison W. Cottrell,et al.  Non-Linear Dimensionality Reduction , 1992, NIPS.

[28]  Jürgen Schmidhuber,et al.  Flat Minima , 1997, Neural Computation.

[29]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[30]  Eric Saund,et al.  A Multiple Cause Mixture Model for Unsupervised Learning , 1995, Neural Computation.

[31]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.