Estimating structured signals in sparse noise: A precise noise sensitivity analysis

We consider the problem of estimating a structured signal xo from linear, underdetermined and noisy measurements y = Ax<sub>0</sub> + z, in the presence of sparse noise z. A natural approach to recovering x<sub>0</sub>, that takes advantage of both the structure of xo and the sparsity of z is solving: x = arg min<sub>x</sub> ||y - Ax||<sub>1</sub> subject to f(x) ≤ f(x<sub>0</sub>) (constrained LAD estimator). Here, f is a convex function aiming to promote the structure of x<sub>0</sub>, say ℓ<sub>1</sub>-norm to promote sparsity or nuclear norm to promote low-rankness. We assume that the entries of A and the non-zero entries of z are i.i.d normal with variances 1 and σ<sup>2</sup>, respectively. Our analysis precisely characterizes the asymptotic noise sensitivity ||x - x<sub>0</sub>||<sup>2</sup><sub>2</sub>/σ<sup>2</sup> in the limit σ<sup>2</sup> → 0. We show analytically that the LAD method outperforms the more popular LASSO method when the noise is sparse. At the same time its performance is no more than π/2 times worse in the presence of non-sparse noise. Our simulation results verify the validity of our theoretical predictions.

[1]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[2]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[3]  Christos Thrampoulidis,et al.  A Tight Version of the Gaussian min-max theorem in the Presence of Convexity , 2014, ArXiv.

[4]  Mihailo Stojnic,et al.  A rigorous geometry-probability equivalence in characterization of ℓ1-optimization , 2013, ArXiv.

[5]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  Babak Hassibi,et al.  Asymptotically Exact Denoising in Relation to Compressed Sensing , 2013, ArXiv.

[7]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[8]  John Wright,et al.  Dense Error Correction Via $\ell^1$-Minimization , 2010, IEEE Transactions on Information Theory.

[9]  Mihailo Stojnic,et al.  Various thresholds for ℓ1-optimization in compressed sensing , 2009, ArXiv.

[10]  Lie Wang The L1L1 penalized LAD estimator for high dimensional linear regression , 2013, J. Multivar. Anal..

[11]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[12]  Trac D. Tran,et al.  Exact Recoverability From Dense Corrupted Observations via $\ell _{1}$-Minimization , 2011, IEEE Transactions on Information Theory.

[13]  MaYi,et al.  Dense error correction via l1-minimization , 2010 .

[14]  D. Sengupta Linear models , 2003 .

[15]  Christos Thrampoulidis,et al.  The squared-error of generalized LASSO: A precise analysis , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[16]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[17]  J. Powell,et al.  Least absolute deviations estimation for the censored regression model , 1984 .

[18]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[19]  Rina Foygel,et al.  Corrupted Sensing: Novel Guarantees for Separating Structured Signals , 2013, IEEE Transactions on Information Theory.

[20]  Christos Thrampoulidis,et al.  Simple Bounds for Noisy Linear Inverse Problems with Exact Side Information , 2013, ArXiv.

[21]  Y. Gordon On Milman's inequality and random subspaces which escape through a mesh in ℝ n , 1988 .

[22]  Joel A. Tropp,et al.  The Bowling Scheme , 2014 .

[23]  Joel A. Tropp,et al.  Living on the edge: A geometric theory of phase transitions in convex optimization , 2013, ArXiv.

[24]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[25]  S. Stigler The Asymptotic Distribution of the Trimmed Mean , 1973 .

[26]  S. R. Searle Linear Models , 1971 .

[27]  R. Vershynin Estimation in High Dimensions: A Geometric Perspective , 2014, 1405.5103.

[28]  Christos Thrampoulidis,et al.  Simple error bounds for regularized noisy linear inverse problems , 2014, 2014 IEEE International Symposium on Information Theory.

[29]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[30]  Yihong Wu,et al.  Shannon Theory for Compressed Sensing , 2011 .

[31]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.

[32]  Mihailo Stojnic,et al.  A framework to characterize performance of LASSO algorithms , 2013, ArXiv.

[33]  D. Pollard Asymptotics for Least Absolute Deviation Regression Estimators , 1991, Econometric Theory.