JUST RELAX: CONVEX PROGRAMMING METHODS FOR SUBSET SELECTION AND SPARSE APPROXIMATION

Subset selection and sparse approximation problems request a good approximation of an input signal using a linear combination of elementary signals, yet they stipulate that the approximation may only involve a few of the elementary signals. This class of problems arises throughout electrical engineering, applied mathematics and statistics, but small theoretical progress has been made over the last fifty years. Subset selection and sparse approximation both admit natural convex relaxations, but the literature contains few results on the behavior of these relaxations for general input signals. This report demonstrates that the solution of the convex program frequently coincides with the solution of the original approximation problem. The proofs depend essentially on geometric properties of the ensemble of elementary signals. The results are powerful because sparse approximation problems are combinatorial, while convex programs can be solved in polynomial time with standard software. Comparable new results for a greedy algorithm, Orthogonal Matching Pursuit, are also stated. This report should have a major practical impact because the theory applies immediately to many real-world signal processing problems.

[1]  E. Schmidt.  Zur Theorie der linearen und nichtlinearen Integralgleichungen , 1907 .

[2]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[3]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[4]  E. Kreyszig.  Introductory Functional Analysis With Applications , 1978 .

[5]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[6]  H. L. Taylor,et al.  Deconvolution with the l 1 norm , 1979 .

[7]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[8]  S. Levy,et al.  Reconstruction of a sparse spike train from a portion of its spectrum and application to high-resolution deconvolution , 1981 .

[9]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[10]  D. Oldenburg,et al.  Recovery of the acoustic impedance from reflection seismograms , 1983 .

[11]  Gene H. Golub,et al.  Matrix computations , 1983 .

[12]  I. Csiszár.  Sanov Property, Generalized $I$-Projection and a Conditional Limit Theorem , 1984 .

[13]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[14]  F. Santosa,et al.  Linear inversion of ban limit reflection seismograms , 1986 .

[15]  N. J. A. Sloane,et al.  Sphere Packings, Lattices and Groups , 1987, Grundlehren der mathematischen Wissenschaften.

[16]  D. Donoho,et al.  Uncertainty principles and signal recovery , 1989 .

[17]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[18]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[19]  A. Atkinson.  Subset Selection in Regression , 1992 .

[20]  Ronald R. Coifman,et al.  Entropy-based algorithms for best basis selection , 1992, IEEE Trans. Inf. Theory.

[21]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[22]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[23]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions , 1994 .

[24]  D. Donoho,et al.  Basis pursuit , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[25]  A. Bos.  Complex gradient and Hessian , 1994 .

[26]  David P. Williamson,et al.  Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming , 1995, JACM.

[27]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[28]  N. J. A. Sloane,et al.  Packing Lines, Planes, etc.: Packings in Grassmannian Spaces , 1996, Exp. Math..

[29]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[30]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[31]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[32]  A. Calderbank,et al.  Z4‐Kerdock Codes, Orthogonal Spreads, and Extremal Euclidean Line‐Sets , 1997 .

[33]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[34]  Marcus J. Grote,et al.  Parallel Preconditioning with Sparse Approximate Inverses , 1997, SIAM J. Sci. Comput..

[35]  Jean-Jacques Fuchs.  Extension of the Pisarenko method to sparse linear arrays , 1997, IEEE Trans. Signal Process..

[36]  I. Johnstone,et al.  Minimax estimation via wavelet shrinkage , 1998 .

[37]  K. Rose.  Deterministic annealing for clustering, compression, classification, regression, and related optimization problems , 1998, Proc. IEEE.

[38]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[39]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[40]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[41]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[42]  Jean-Jacques Fuchs,et al.  Detection and estimation of superimposed signals , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[43]  Bhaskar D. Rao,et al.  An affine scaling methodology for best basis selection , 1999, IEEE Trans. Signal Process..

[44]  Chee-Keng Yap,et al.  Fundamental problems of algorithmic algebra , 1999 .

[45]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[46]  J. Rohn.  Computing the norm ∥A∥∞,1 is NP-hard , 2000 .

[47]  Yoram Bresler,et al.  On the Optimality of the Backward Greedy Algorithm for the Subset Selection Problem , 2000, SIAM J. Matrix Anal. Appl..

[48]  P. Tseng,et al.  Block Coordinate Relaxation Methods for Nonparametric Wavelet Denoising , 2000 .

[49]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[50]  Michael Elad,et al.  A generalized uncertainty principle and sparse representation in pairs of bases , 2002, IEEE Trans. Inf. Theory.

[51]  N.J.A. Sloane,et al.  Packing Planes in Four Dimensions and Other Mysteries , 2002, math/0208017.

[52]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[53]  V. Temlyakov.  Nonlinear Methods of Approximation , 2003, Found. Comput. Math..

[54]  S. Muthukrishnan,et al.  Approximation of functions over redundant dictionaries using coherence , 2003, SODA '03.

[55]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[56]  Rémi Gribonval,et al.  Harmonic decomposition of audio signals with matching pursuit , 2003, IEEE Trans. Signal Process..

[57]  E. Candès,et al.  Astronomical image representation by the curvelet transform , 2003, Astronomy & Astrophysics.

[58]  Avideh Zakhor,et al.  Matching pursuits based multiple description video coding for lossy environments , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[59]  Thomas Strohmer,et al.  GRASSMANNIAN FRAMES WITH APPLICATIONS TO CODING AND COMMUNICATION , 2003, math/0301135.

[60]  S. Muthukrishnan,et al.  Improved sparse approximation over quasiincoherent dictionaries , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[61]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[62]  Arkadi Nemirovski,et al.  On sparse representation in pairs of bases , 2003, IEEE Trans. Inf. Theory.

[63]  Pascal Frossard,et al.  A posteriori quantization of progressive matching pursuit streams , 2004, IEEE Transactions on Signal Processing.

[64]  Joel A. Tropp,et al.  Topics in sparse approximation , 2004 .

[65]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[66]  Pedro M. Domingos.  The Role of Occam's Razor in Knowledge Discovery , 1999, Data Mining and Knowledge Discovery.

[67]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[68]  J. Tropp.  Recovery of short, complex linear combinations via 𝓁1 minimization , 2005, IEEE Trans. Inf. Theory.

[69]  Joel A. Tropp,et al.  Recovery of short, complex linear combinations via /spl lscr//sub 1/ minimization , 2005, IEEE Transactions on Information Theory.

[70]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[71]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[72]  Dany Leviatan,et al.  Simultaneous approximation by greedy algorithms , 2006, Adv. Comput. Math..