Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems

Many problems in signal processing and statistical inference involve finding sparse solutions to under-determined, or ill-conditioned, linear systems of equations. A standard approach consists in minimizing an objective function which includes a quadratic (squared ) error term combined with a sparseness-inducing regularization term. Basis pursuit, the least absolute shrinkage and selection operator (LASSO), wavelet-based deconvolution, and compressed sensing are a few well-known examples of this approach. This paper proposes gradient projection (GP) algorithms for the bound-constrained quadratic programming (BCQP) formulation of these problems. We test variants of this approach that select the line search parameters in different ways, including techniques based on the Barzilai-Borwein method. Computational experiments show that these GP approaches perform well in a wide range of applications, often being significantly faster (in terms of computation time) than competing methods. Although the performance of GP methods tends to degrade as the regularization term is de-emphasized, we show how they can be embedded in a continuation scheme to recover their efficient practical performance.

[1]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[2]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[3]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[4]  H. L. Taylor,et al.  Deconvolution with the l 1 norm , 1979 .

[5]  S. Levy,et al.  Reconstruction of a sparse spike train from a portion of its spectrum and application to high-resolution deconvolution , 1981 .

[6]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[7]  F. Santosa,et al.  Linear inversion of ban limit reflection seismograms , 1986 .

[8]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[9]  Stephen J. Wright Implementing proximal point methods for linear programming , 1990 .

[10]  Gerardo Toraldo,et al.  On the Solution of Large Quadratic Programming Problems with Bound Constraints , 1991, SIAM J. Optim..

[11]  A. Atkinson Subset Selection in Regression , 1992 .

[12]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[13]  Stefano Alliney,et al.  An algorithm for the minimization of mixed l1 and l2 norms with application to Bayesian estimation , 1994, IEEE Trans. Signal Process..

[14]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1995 .

[15]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[16]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[17]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[18]  Stephen J. Wright Primal-Dual Interior-Point Methods , 1997, Other Titles in Applied Mathematics.

[19]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[20]  S. Mallat A wavelet tour of signal processing , 1998 .

[21]  Jean-Jacques Fuchs,et al.  Multipath time-delay detection and estimation , 1999, IEEE Trans. Signal Process..

[22]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[23]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[24]  R. Nowak,et al.  Fast wavelet-based image deconvolution using the EM algorithm , 2001, Conference Record of Thirty-Fifth Asilomar Conference on Signals, Systems and Computers (Cat.No.01CH37256).

[25]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[26]  Stephen J. Wright,et al.  Warm-Start Strategies in Interior-Point Methods for Linear Programming , 2002, SIAM J. Optim..

[27]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[28]  J. Fuchs More on sparse representations in arbitrary bases , 2003 .

[29]  A. Iusem On the convergence properties of the projected gradient method for convex optimization , 2003 .

[30]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[31]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[32]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[33]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[34]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[35]  M. Nikolova An Algorithm for Total Variation Minimization and Applications , 2004 .

[36]  J. Tropp JUST RELAX: CONVEX PROGRAMMING METHODS FOR SUBSET SELECTION AND SPARSE APPROXIMATION , 2004 .

[37]  Roger Fletcher,et al.  Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming , 2005, Numerische Mathematik.

[38]  Shai Avidan,et al.  Spectral Bounds for Sparse PCA: Exact and Greedy Algorithms , 2005, NIPS.

[39]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[40]  Robert D. Nowak,et al.  A bound optimization approach to wavelet-based image deconvolution , 2005, IEEE International Conference on Image Processing 2005.

[41]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[42]  Stephen J. Wright,et al.  Simultaneous Variable Selection , 2005, Technometrics.

[43]  Luca Zanni,et al.  Gradient projection methods for quadratic programs and applications in training support vector machines , 2005, Optim. Methods Softw..

[44]  Berwin A. Turlach,et al.  On algorithms for solving least squares problems under an L1 penalty or an L1 constraint , 2005 .

[45]  Dmitry M. Malioutov,et al.  Homotopy continuation for sparse signal representation , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[46]  Robert D. Nowak,et al.  Signal Reconstruction From Noisy Random Projections , 2006, IEEE Transactions on Information Theory.

[47]  Michael Elad,et al.  Image Denoising with Shrinkage and Redundant Representations , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[48]  Harald Haas,et al.  Asilomar Conference on Signals, Systems, and Computers , 2006 .

[49]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[50]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[51]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[52]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[53]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[54]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[55]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[56]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[57]  Jacek Gondzio,et al.  A New Unblocking Technique to Warmstart Interior Point Methods Based on Sensitivity Analysis , 2008, SIAM J. Optim..

[58]  E. Alper Yildirim,et al.  Implementation of warm-start strategies in interior-point methods for linear programming in fixed dimension , 2008, Comput. Optim. Appl..

[59]  Mike E. Davies,et al.  Gradient Pursuits , 2008, IEEE Transactions on Signal Processing.

[60]  Richard W. Cottle,et al.  Linear Complementarity Problem , 2009, Encyclopedia of Optimization.