On the convergence of FOCUSS algorithm for sparse representation

FOCal Underdetermined System Solver (FOCUSS) is a powerful tool for sparse representation and underdetermined inverse problems, which is extremely easy to implement. In this paper, we provide a comprehensive convergence analysis on the FOCUSS algorithm towards establishing a systematic convergence theory for it. First, we give a rigorous derivation for this algorithm exploiting the auxiliary function. Then, we prove its convergence. In particular, we systematically analyze its convergence rate for different sparsity parameter p and demonstrate its convergence rate by numerical experiments.

[1]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[2]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[3]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[4]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[5]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[6]  Paul Tseng.  Further Results on Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise , 2009, IEEE Transactions on Information Theory.

[7]  Yonina C. Eldar,et al.  Blind Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[8]  Bhaskar D. Rao,et al.  Signal processing with the sparseness constraint , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[9]  Rémi Gribonval,et al.  Restricted Isometry Constants Where $\ell ^{p}$ Sparse Recovery Can Fail for $0≪ p \leq 1$ , 2009, IEEE Transactions on Information Theory.

[10]  Mark D. Plumbley.  On Polar Polytopes and the Recovery of Sparse Representations , 2007, IEEE Trans. Inf. Theory.

[11]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[12]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[13]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[14]  Irina F. Gorodnitsky.  An extension of an interior-point method for entropy minimization , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[15]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[16]  Bhaskar D. Rao,et al.  An affine scaling methodology for best basis selection , 1999, IEEE Trans. Signal Process..

[17]  Michael I. Jordan,et al.  On Convergence Properties of the EM Algorithm for Gaussian Mixtures , 1996, Neural Computation.

[18]  Brendt Wohlberg,et al.  Noise sensitivity of sparse signal representations: reconstruction error bounds for the inverse problem , 2003, IEEE Trans. Signal Process..

[19]  Martin J. McKeown,et al.  Underdetermined Anechoic Blind Source Separation via $\ell^{q}$-Basis-Pursuit With $q≪1$ , 2007, IEEE Transactions on Signal Processing.

[20]  Davies Rémi Gribonval.  Restricted Isometry Constants Where Lp Sparse Recovery Can Fail for 0 , 2008 .

[21]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[22]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[23]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[24]  Jie Chen,et al.  Theoretical Results on Sparse Representations of Multiple-Measurement Vectors , 2006, IEEE Transactions on Signal Processing.

[25]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[26]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[27]  J.J. Fuchs,et al.  Convergence of a Sparse Representations Algorithm Applicable to Real or Complex Data , 2007, IEEE Journal of Selected Topics in Signal Processing.

[28]  Zhaoshui He,et al.  Improved FOCUSS Method With Conjugate Gradient Iterations , 2009, IEEE Transactions on Signal Processing.

[29]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[30]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[31]  Pierre Vandergheynst,et al.  On the exponential convergence of matching pursuits in quasi-incoherent dictionaries , 2006, IEEE Transactions on Information Theory.

[32]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[33]  Bhaskar D. Rao,et al.  Subset selection in noise based on diversity measure minimization , 2003, IEEE Trans. Signal Process..

[34]  D. Donoho,et al.  Fast Solution of -Norm Minimization Problems When the Solution May Be Sparse , 2008 .

[35]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[36]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[37]  K. Kreutz-Delgado,et al.  Deriving algorithms for computing sparse solutions to linear inverse problems , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[38]  Jean-Jacques Fuchs,et al.  Recovery of exact sparse representations in the presence of bounded noise , 2005, IEEE Transactions on Information Theory.

[39]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[40]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[41]  Mineichi Kudo,et al.  Performance analysis of minimum /spl lscr//sub 1/-norm solutions for underdetermined source separation , 2004, IEEE Transactions on Signal Processing.

[42]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .