On the Error of Estimating the Sparsest Solution of Underdetermined Linear Systems

Let A be an n × m matrix with m >; n, and suppose that the underdetermined linear system As = x admits a sparse solution S<sub>0</sub> for which ||S<sub>0</sub>||<sub>0</sub> <; 1/2 spark( A). Such a sparse solution is unique due to a well-known uniqueness theorem. Suppose now that we have somehow a solution ŝ as an estimation of s<sub>0</sub>, and suppose that ŝ is only "approximately sparse," that is, many of its components are very small and nearly zero, but not mathematically equal to zero. Is such a solution necessarily close to the true sparsest solution? More generally, is it possible to construct an upper bound on the estimation error ||ŝ - s<sub>0</sub>||<sub>2</sub> without knowing S<sub>0</sub>? The answer is positive, and in this paper, we construct such a bound based on minimal singular values of submatrices of A. We will also state a tight bound, which is more complicated, but besides being tight, enables us to study the case of random dictionaries and obtain probabilistic upper bounds. We will also study the noisy case, that is, where x = As + n. Moreover, we will see that where ||s<sub>0</sub> ||<sub>0</sub> grows, to obtain a predetermined guaranty on the maximum of ||ŝ - s<sub>0</sub> ||<sub>2</sub>, ŝ is needed to be sparse with a better approximation. This can be seen as an explanation to the fact that the estimation quality of sparse recovery algorithms degrades where ||s<sub>0</sub>||<sub>0</sub> grows.

[1]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[2]  Emmanuel J. Cand The Restricted Isometry Property and Its Implications for Compressed Sensing , 2008 .

[3]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[4]  Z. Bai,et al.  On the limit of the largest eigenvalue of the large dimensional sample covariance matrix , 1988 .

[5]  Robert D. Nowak,et al.  A bound optimization approach to wavelet-based image deconvolution , 2005, IEEE International Conference on Image Processing 2005.

[6]  Jianhong Shen On the singular values of Gaussian random matrices , 2001 .

[7]  S. Foucart,et al.  Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 , 2009 .

[8]  Christian Jutten,et al.  A Fast Method for Sparse Component Analysis Based on Iterative Detection‐Estimation , 2006 .

[9]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[10]  Pierre Vandergheynst,et al.  A simple test to check the optimality of a sparse signal approximation , 2006, Signal Process..

[11]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[12]  S. Geman A Limit Theorem for the Norm of Random Matrices , 1980 .

[13]  S. Amari,et al.  SPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES , 2003 .

[14]  R.G. Baraniuk,et al.  Compressive Sensing [Lecture Notes] , 2007, IEEE Signal Processing Magazine.

[15]  S. Szarek,et al.  Chapter 8 - Local Operator Theory, Random Matrices and Banach Spaces , 2001 .

[16]  Jean-Jacques Fuchs,et al.  Recovery of exact sparse representations in the presence of bounded noise , 2005, IEEE Transactions on Information Theory.

[17]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[18]  Christian Jutten,et al.  A Fast Approach for Overcomplete Sparse Decomposition Based on Smoothed $\ell ^{0}$ Norm , 2008, IEEE Transactions on Signal Processing.

[19]  Wotao Yin,et al.  Sparse Signal Reconstruction via Iterative Support Detection , 2009, SIAM J. Imaging Sci..

[20]  Christian Jutten,et al.  An upper bound on the estimation error of the sparsest solution of underdetermined linear systems , 2009 .

[21]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[22]  Rémi Gribonval,et al.  Restricted Isometry Constants Where $\ell ^{p}$ Sparse Recovery Can Fail for $0≪ p \leq 1$ , 2009, IEEE Transactions on Information Theory.

[23]  Christian Jutten,et al.  On the Stable Recovery of the Sparsest Overcomplete Representations in Presence of Noise , 2010, IEEE Transactions on Signal Processing.

[24]  E. Candès The restricted isometry property and its implications for compressed sensing , 2008 .

[25]  J. Tropp Algorithms for simultaneous sparse approximation. Part II: Convex relaxation , 2006, Signal Process..

[26]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[27]  Brendt Wohlberg,et al.  Noise sensitivity of sparse signal representations: reconstruction error bounds for the inverse problem , 2003, IEEE Trans. Signal Process..

[28]  Rachel Ward,et al.  Compressed Sensing With Cross Validation , 2008, IEEE Transactions on Information Theory.

[29]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[30]  Christian Jutten,et al.  Fast Sparse Representation Based on Smoothed l0 Norm , 2007, ICA.

[31]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[32]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[33]  A. Edelman Eigenvalues and condition numbers of random matrices , 1988 .

[34]  Dmitry M. Malioutov,et al.  Sequential Compressed Sensing , 2010, IEEE Journal of Selected Topics in Signal Processing.

[35]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[36]  Rémi Gribonval,et al.  A survey of Sparse Component Analysis for blind source separation: principles, perspectives, and new challenges , 2006, ESANN.

[37]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[38]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[39]  Christian Jutten,et al.  Robust-SL0 for stable sparse representation in noisy settings , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[40]  A TroppJoel Corrigendum in "Just relax: Convex programming methods for identifying sparse signals in noise" , 2009 .

[41]  Davies Rémi Gribonval Restricted Isometry Constants Where Lp Sparse Recovery Can Fail for 0 , 2008 .

[42]  J. W. Silverstein The Smallest Eigenvalue of a Large Dimensional Wishart Matrix , 1985 .

[43]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[44]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[45]  J. Tropp Corrigendum in “Just Relax: Convex Programming Methods for Identifying Sparse Signals in Noise” [Mar 06 1030-1051] , 2009 .

[46]  Z. Bai,et al.  Limit of the smallest eigenvalue of a large dimensional sample covariance matrix , 1993 .

[47]  Malik Magdon-Ismail,et al.  On selecting a maximum volume sub-matrix of a matrix and related problems , 2009, Theor. Comput. Sci..

[48]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[49]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[50]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..