Neurally plausible sparse coding via thresholding and local competition

While evidence indicates that neural systems may be employing sparse approximations to represent sensed stimuli, the mechanisms underlying this ability are not understood. We present a class of neurally plausible locally competitive algorithms (LCAs) that correspond to a collection of sparse coding principles minimizing a weighted combination of mean-squared error (MSE) and a coe! cient cost function. LCAs use thresholding functions to induce local (usually one-way) inhibitory competitions. In contrast to greedy algorithms that iteratively select the single best element, LCAs allow continual interaction among many units. LCAs produce coe! cients with sparsity levels comparable to existing sparse coding algorithms while being plausible for neural implementation. Additionally, LCAs coe! cients for video sequences demonstrate inertial properties that are both qualitatively and quantitatively more regular (i.e., smoother and more predictable) than the coe! cients produced by greedy algorithms.

[1]  Bruno A Olshausen,et al.  Sparse coding of sensory inputs , 2004, Current Opinion in Neurobiology.

[2]  P. O. Bishop,et al.  Spatial vision. , 1971, Annual review of psychology.

[3]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[4]  A. Morse,et al.  Stability of switched systems with average dwell-time , 1999, Proceedings of the 38th IEEE Conference on Decision and Control (Cat. No.99CH36304).

[5]  Nick G. Kingsbury,et al.  Redundant representation with complex wavelets: how to achieve sparsity , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[6]  William T. Freeman,et al.  Presented at: 2nd Annual IEEE International Conference on Image , 1995 .

[7]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[8]  O. Christensen An introduction to frames and Riesz bases , 2002 .

[9]  David L. Donoho,et al.  Neighborly Polytopes And Sparse Solution Of Underdetermined Linear Equations , 2005 .

[10]  Bruno A. Olshausen,et al.  Book Review , 2003, Journal of Cognitive Neuroscience.

[11]  Joel A. Tropp,et al.  Sparse Approximation Via Iterative Thresholding , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[12]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[13]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[14]  Michael S. Lewicki,et al.  Efficient coding of natural sounds , 2002, Nature Neuroscience.

[15]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[16]  T. Berger Rate-Distortion Theory , 2003 .

[17]  Simon J. Thorpe,et al.  Coding static natural images using spiking event times: do neurons Cooperate? , 2004, IEEE Transactions on Neural Networks.

[18]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions with matching pursuit , 1994, Defense, Security, and Sensing.

[19]  R. Decarlo,et al.  Perspectives and results on the stability and stabilizability of hybrid systems , 2000, Proceedings of the IEEE.

[20]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[21]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[22]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[23]  E. Candès,et al.  New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities , 2004 .

[25]  Martin Rehn,et al.  A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields , 2007, Journal of Computational Neuroscience.

[26]  D. Vere-Jones Markov Chains , 1972, Nature.

[27]  Wolfgang Porod,et al.  Qualitative analysis and synthesis of a class of neural networks , 1988 .

[28]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[29]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[30]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[31]  Nicolai Petkov,et al.  Fast atomic decomposition by the inhibition method , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[32]  Tharam S. Dillon,et al.  Exponential stability and oscillation of Hopfield graded response neural network , 1994, IEEE Trans. Neural Networks.

[33]  Simon J. Thorpe,et al.  Sparse spike coding in an asynchronous feed-forward multi-layer neural network using matching pursuit , 2004, Neurocomputing.

[34]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[35]  Gene F. Franklin,et al.  Feedback Control of Dynamic Systems , 1986 .

[36]  Gabriel Cristóbal,et al.  Sparse overcomplete Gabor wavelet representation based on local competitions , 2006, IEEE Transactions on Image Processing.

[37]  A. Bacciotti,et al.  Liapunov functions and stability in control theory , 2001 .

[38]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[39]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[40]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[41]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[42]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[43]  L. Rebollo-Neira,et al.  Optimized orthogonal matching pursuit approach , 2002, IEEE Signal Processing Letters.

[44]  Andreas Turk,et al.  Hierarchical parallel matching pursuit , 1994, Optics & Photonics.

[45]  B. Delgutte,et al.  Neural coding of the temporal envelope of speech : Relation to modulation transfer functions , 2001 .

[46]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[47]  J. Gallant,et al.  Natural Stimulation of the Nonclassical Receptive Field Increases Information Transmission Efficiency in V1 , 2002, The Journal of Neuroscience.