Estimation of Sparse Nonnegative Sources from Noisy Overcomplete Mixtures Using MAP

In this letter, we propose a new algorithm for estimating sparse nonnegative sources from a set of noisy linear mixtures. In particular, we consider difficult situations with high noise levels and more sources than sensors (underdetermined case). We show that when sources are very sparse in time and overlapped at some locations, they can be recovered even with very low signal-to-noise ratio, and by using many fewer sensors than sources. A theoretical analysis based on Bayesian estimation tools is included showing strong connections with algorithms in related areas of research such as ICA, NMF, FOCUSS, and sparse representation of data with overcomplete dictionaries. Our algorithm uses a Bayesian approach by modeling sparse signals through mixed-state random variables. This new model for priors imposes 0 norm-based sparsity. We start our analysis for the case of nonoverlapped sources (1-sparse), which allows us to simplify the search of the posterior maximum avoiding a combinatorial search. General algorithms for overlapped cases, such as 2-sparse and k-sparse sources, are derived by using the algorithm for 1-sparse signals recursively. Additionally, a combination of our MAP algorithm with the NN-KSVD algorithm is proposed for estimating the mixing matrix and the sources simultaneously in a real blind fashion. A complete set of simulation results is included showing the performance of our algorithm.

[1]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[2]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[3]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[4]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[5]  B. Moor,et al.  Independent component analysis based on higher-order statistics only , 1996, Proceedings of 8th Workshop on Statistical Signal and Array Processing.

[6]  C. Robert.  The Bayesian choice : a decision-theoretic motivation , 1996 .

[7]  William B. Levy,et al.  Energy Efficient Neural Codes , 1996, Neural Computation.

[8]  H. Kaneko,et al.  Classifying neuronal spikes from multiunit recording by using a multisite electrode , 1996, Proceedings of 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  G D Lewen,et al.  Reproducibility and Variability in Neural Spike Trains , 1997, Science.

[10]  Rob R. de Ruyter van Steveninck,et al.  The metabolic cost of neural information , 1998, Nature Neuroscience.

[11]  K. Kreutz-Delgado,et al.  Basis selection in the presence of noise , 1998, Conference Record of Thirty-Second Asilomar Conference on Signals, Systems and Computers (Cat. No.98CH36284).

[12]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[13]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[14]  Kevin H. Knuth.  A Bayesian approach to source separation , 1999 .

[15]  J L Gallant,et al.  Sparse coding and decorrelation in primary visual cortex during natural vision. , 2000, Science.

[16]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[17]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[18]  Allan Kardec Barros,et al.  The Independence Assumption: Dependent Component Analysis , 2000 .

[19]  J. Krtiger.  Multimicroelectrode Investigation of Monkey Striate Cortex : Spike Train Correlations in the Infragranular Layers , 2001 .

[20]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[21]  Wulfram Gerstner,et al.  Spiking Neuron Models , 2002 .

[22]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[23]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[24]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[25]  Andrzej Cichocki,et al.  A robust approach to independent component analysis of signals with high-level noise measurements , 2003, IEEE Trans. Neural Networks.

[26]  M. DeWeese,et al.  Binary Spiking in Auditory Cortex , 2003, The Journal of Neuroscience.

[27]  K. Kreutz-Delgado,et al.  Sparse image coding using learned overcomplete dictionaries , 2004, Proceedings of the 2004 14th IEEE Signal Processing Society Workshop Machine Learning for Signal Processing, 2004..

[28]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[29]  Ali Mohammad-Djafari,et al.  A Bayesian method for positive source separation , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[30]  M. Davies,et al.  Identifiability issues in noisy ICA , 2004, IEEE Signal Processing Letters.

[31]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[32]  Michael Elad,et al.  K-SVD and its non-negative variant for dictionary design , 2005, SPIE Optics + Photonics.

[33]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[34]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[35]  Cesar F. Caiafa,et al.  Separation of statistically dependent sources using an L2-distance non-Gaussianity measure , 2006, Signal Process..

[36]  C. Caiafa,et al.  Separation of statistically dependent sources using an L 2 -distance non-Gaussianity measure , 2006 .

[37]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[38]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[39]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[40]  Ali Mohammad-Djafari,et al.  Bayesian source separation: beyond PCA and ICA , 2006, ESANN.

[41]  C. Yoo,et al.  Underdetermined Blind Source Separation Based on Generalized Gaussian Distribution , 2006, 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing.

[42]  D. Brie,et al.  Separation of Non-Negative Mixture of Non-Negative Sources Using a Bayesian Approach and MCMC Sampling , 2006, IEEE Transactions on Signal Processing.

[43]  Andrzej Cichocki,et al.  New Algorithms for Non-Negative Matrix Factorization in Applications to Blind Source Separation , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[44]  Patrick Bouthemy,et al.  Mixed-State Auto-Models and Motion Texture Modeling , 2006, Journal of Mathematical Imaging and Vision.

[45]  Ali Mohammad-Djafari,et al.  Hidden Markov models for wavelet-based blind source separation , 2006, IEEE Transactions on Image Processing.

[46]  Florian Steinke,et al.  Bayesian Inference and Optimal Design in the Sparse Linear Model , 2007, AISTATS.

[47]  Michael S. Lewicki,et al.  Robust Coding Over Noisy Overcomplete Channels , 2007, IEEE Transactions on Image Processing.

[48]  Bhaskar D. Rao,et al.  An Empirical Bayesian Strategy for Solving the Simultaneous Sparse Approximation Problem , 2007, IEEE Transactions on Signal Processing.

[49]  A. Cichocki,et al.  Flexible HALS algorithms for sparse non-negative matrix/tensor factorization , 2008, 2008 IEEE Workshop on Machine Learning for Signal Processing.

[50]  Liqing Zhang,et al.  A Note on Lewicki-Sejnowski Gradient for Learning Overcomplete Representations , 2008, Neural Computation.

[51]  Arvind Kumar,et al.  The High-Conductance State of Cortical Networks , 2008, Neural Computation.

[52]  S. Amari,et al.  Nonnegative Matrix and Tensor Factorization [Lecture Notes] , 2008, IEEE Signal Processing Magazine.

[53]  A. Bruckstein,et al.  Sparse non-negative solution of a linear system of equations is unique , 2008, 2008 3rd International Symposium on Communications, Control and Signal Processing.

[54]  Nancy Bertin,et al.  Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis , 2009, Neural Computation.

[55]  David P. Wipf,et al.  A unified Bayesian framework for MEG/EEG source imaging , 2009, NeuroImage.

[56]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .