An Iterative Bayesian Algorithm for Sparse Component Analysis in Presence of Noise

We present a Bayesian approach for sparse component analysis (SCA) in the noisy case. The algorithm is essentially a method for obtaining sufficiently sparse solutions of underdetermined systems of linear equations with additive Gaussian noise. In general, an underdetermined system of linear equations has infinitely many solutions. However, it has been shown that sufficiently sparse solutions can be uniquely identified. Our main objective is to find this unique solution. Our method is based on a novel estimation of source parameters and maximum a posteriori (MAP) estimation of sources. To tackle the great complexity of the MAP algorithm (when the number of sources and mixtures become large), we propose an iterative Bayesian algorithm (IBA). This IBA algorithm is based on the MAP estimation of sources, too, but optimized with a steepest-ascent method. The convergence analysis of the IBA algorithm and its convergence to true global maximum are also proved. Simulation results show that the performance achieved by the IBA algorithm is among the best, while its complexity is rather high in comparison to other algorithms. Simulation results also show the low sensitivity of the IBA algorithm to its simulation parameters.

[1]  Lawrence Carin,et al.  Bayesian Compressive Sensing , 2008, IEEE Transactions on Signal Processing.

[2]  Donald B. Rubin,et al.  Max-imum Likelihood from Incomplete Data , 1972 .

[3]  Ali Mohammad-Djafari,et al.  Bayesian source separation: beyond PCA and ICA , 2006, ESANN.

[4]  Christian Jutten,et al.  Sparse ICA via cluster-wise PCA , 2006, Neurocomputing.

[5]  Erik G. Larsson,et al.  Linear Regression With a Sparse Parameter Vector , 2006, IEEE Transactions on Signal Processing.

[6]  Jerry M. Mendel,et al.  Maximum-Likelihood Deconvolution , 1989 .

[7]  Michael Elad,et al.  Low Bit-Rate Compression of Facial Images , 2007, IEEE Transactions on Image Processing.

[8]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[9]  L. Vielva,et al.  UNDERDETERMINED BLIND SOURCE SEPARATION USING A PROBABILISTIC SOURCE SPARSITY MODEL , 2001 .

[10]  N. Mitianoudis,et al.  Simple mixture model for sparse overcomplete ICA , 2004 .

[11]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[12]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[13]  Simon J. Godsill,et al.  A Bayesian Approach for Blind Separation of Sparse Sources , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[14]  Christian Jutten,et al.  Sparse Component Analysis in Presence of Noise Using an Iterative EM-MAP Algorithm , 2007, ICA.

[15]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[16]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[17]  C. Jutten,et al.  Source Estimation in Noisy Sparse Component Analysis , 2007, 2007 15th International Conference on Digital Signal Processing.

[18]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[19]  Christian Jutten,et al.  A Fast Approach for Overcomplete Sparse Decomposition Based on Smoothed $\ell ^{0}$ Norm , 2008, IEEE Transactions on Signal Processing.

[20]  Onur G. Guleryuz,et al.  Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part I: theory , 2006, IEEE Transactions on Image Processing.

[21]  Barak A. Pearlmutter,et al.  Blind source separation by sparse decomposition , 2000, SPIE Defense + Commercial Sensing.

[22]  JuttenChristian,et al.  A fast approach for overcomplete sparse decomposition based on smoothed l0 norm , 2009 .

[23]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[24]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[25]  Mike E. Davies,et al.  Compressed Sensing and Source Separation , 2007, ICA.

[26]  L. Vandendorpe,et al.  Oversampled filter banks as error correcting codes: theory and impulse noise correction , 2005, IEEE Transactions on Signal Processing.

[27]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[28]  Fabian J. Theis,et al.  Sparse component analysis and blind source separation of underdetermined mixtures , 2005, IEEE Transactions on Neural Networks.

[29]  Christian Jutten,et al.  A Fast Method for Sparse Component Analysis Based on Iterative Detection‐Estimation , 2006 .

[30]  Christian Jutten,et al.  Fast Sparse Representation Based on Smoothed l0 Norm , 2007, ICA.

[31]  Sacha Krstulovic,et al.  Mptk: Matching Pursuit Made Tractable , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[32]  Christian Jutten,et al.  Fast Sparse Representation based on Smoothed � , 2006 .

[33]  Michael Elad,et al.  Submitted to Ieee Transactions on Image Processing Image Decomposition via the Combination of Sparse Representations and a Variational Approach , 2022 .

[34]  Marc Lavielle,et al.  Bayesian deconvolution of Bernoulli-Gaussian processes , 1993, Signal Process..

[35]  Vahid Tarokh,et al.  On Sparsity, Redundancy and Quality of Frame Representations , 2007, 2007 IEEE International Symposium on Information Theory.

[36]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[37]  Rémi Gribonval,et al.  A survey of Sparse Component Analysis for blind source separation: principles, perspectives, and new challenges , 2006, ESANN.

[38]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[39]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[40]  Antonio Artés-Rodríguez,et al.  Sparse deconvolution using adaptive mixed-Gaussian models , 1996, Signal Process..

[41]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[42]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[43]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[44]  Luc Vandendorpe,et al.  Oversampled filter banks as error correcting codes , 2002, The 5th International Symposium on Wireless Personal Multimedia Communications.

[45]  Mike E. Davies,et al.  Gradient Pursuits , 2008, IEEE Transactions on Signal Processing.

[46]  Jerry M. Mendel,et al.  Maximum likelihood detection and estimation of Bernoulli - Gaussian processes , 1982, IEEE Trans. Inf. Theory.

[47]  E. Candes,et al.  11-magic : Recovery of sparse signals via convex programming , 2005 .

[48]  Christian Jutten,et al.  Decoding real-field codes by an iterative Expectation-Maximization (EM) algorithm , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[49]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[50]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[51]  H. L. Taylor,et al.  Deconvolution with the l 1 norm , 1979 .

[52]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.