Lococode Performs Nonlinear ICA Without Knowing The Number Of Sources

Low-complexity coding and decoding (Lococode), a novel approach to sensory coding, trains autoassocia-tors (AAs) by Flat Minimum Search (FMS), a recent general method for nding low-complexity networks with high generalization capability. FMS works by minimizing both training error and required weight precision. We nd that as a by-product Lococode separates nonlinear superpositions of sources without knowing their number. Assuming that the input data can be reduced to few simple causes (this is often the case with visual data), according to our theoretical analysis the hidden layer of an FMS-trained AA tends to code each input by a sparse code based on as few simple, independent features as possible. In experiments Lo-cocode extracts optimal codes for diicult, nonlinear versions of the \noisy bars" benchmark problem, while traditional ICA and PCA do not.

[1]  H. B. Barlow,et al.  Finding Minimum Entropy Codes , 1989, Neural Computation.

[2]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[3]  M. Mozer.  Discovering Discrete Distributed Representations with Iterative Competitive Learning , 1990, NIPS 1990.

[4]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[5]  Jürgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992, Neural Computation.

[6]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[7]  Geoffrey E. Hinton,et al.  Developing Population Codes by Minimizing Description Length , 1993, NIPS.

[8]  J. Nadal.  Non linear neurons in the low noise limit : a factorial code maximizes information transferJean , 1994 .

[9]  Jürgen Schmidhuber,et al.  Simplifying Neural Nets by Discovering Flat Minima , 1994, NIPS.

[10]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[11]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[12]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[13]  Gustavo Deco,et al.  Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures , 1995, Neural Networks.

[14]  R. Zemel,et al.  Learning sparse multiple cause models , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[15]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[16]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[17]  Jürgen Schmidhuber,et al.  Semilinear Predictability Minimization Produces Well-Known Feature Detectors , 1996, Neural Computation.

[18]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[19]  Geoffrey E. Hinton,et al.  Generative models for discovering sparse distributed representations. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[20]  Bruno A. Olshausen,et al.  Inferring Sparse, Overcomplete Image Codes Using an Efficient Coding Framework , 1998, NIPS.

[21]  Jürgen Schmidhuber,et al.  Flat Minima , 1997, Neural Computation.

[22]  Peter Földiák,et al.  Sparse coding in the primate cortex , 1998 .

[23]  Ali Mansour,et al.  Blind Separation of Sources , 1999 .