Convergence Analysis of the FOCUSS Algorithm

Focal Underdetermined System Solver (FOCUSS) is a powerful and easy to implement tool for basis selection and inverse problems. One of the fundamental problems regarding this method is its convergence, which remains unsolved until now. We investigate the convergence of the FOCUSS algorithm in this paper. We first give a rigorous derivation for the FOCUSS algorithm by exploiting the auxiliary function. Following this, we further prove its convergence by stability analysis.

[1]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[2]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[3]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[4]  Irina F. Gorodnitsky.  An extension of an interior-point method for entropy minimization , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[5]  Brendt Wohlberg,et al.  Noise sensitivity of sparse signal representations: reconstruction error bounds for the inverse problem , 2003, IEEE Trans. Signal Process..

[6]  Jean-Jacques Fuchs,et al.  Recovery of exact sparse representations in the presence of bounded noise , 2005, IEEE Transactions on Information Theory.

[7]  J. C. Ye,et al.  Projection reconstruction MR imaging using FOCUSS , 2007, Magnetic resonance in medicine.

[8]  Jong Chul Ye,et al.  Improved k–t BLAST and k–t SENSE using FOCUSS , 2007, Physics in medicine and biology.

[9]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[10]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[11]  Andrzej Cichocki,et al.  Improved M-FOCUSS Algorithm With Overlapping Blocks for Locally Smooth Sparse Signals , 2008, IEEE Transactions on Signal Processing.

[12]  Guillaume Obozinski,et al.  Sparse methods for machine learning Theory and algorithms , 2012 .

[13]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[14]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[17]  Shengli Xie,et al.  Blind Spectral Unmixing Based on Sparse Nonnegative Matrix Factorization , 2011, IEEE Transactions on Image Processing.

[18]  Zongben Xu,et al.  $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Ran He,et al.  Two-Stage Nonnegative Sparse Representation for Large-Scale Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Bhaskar D. Rao,et al.  An affine scaling methodology for best basis selection , 1999, IEEE Trans. Signal Process..

[21]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[22]  Jun Wang,et al.  An Improved Algebraic Criterion for Global Exponential Stability of Recurrent Neural Networks With Time-Varying Delays , 2008, IEEE Transactions on Neural Networks.

[23]  S. Rangan,et al.  Orthogonal Matching Pursuit from Noisy Measurements : A New Analysis ∗ , 2009 .

[24]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[25]  Huaguang Zhang,et al.  Global Asymptotic Stability of Recurrent Neural Networks With Multiple Time-Varying Delays , 2008, IEEE Transactions on Neural Networks.

[26]  Shengli Xie,et al.  Online Blind Source Separation Using Incremental Nonnegative Matrix Factorization With Volume Constraint , 2011, IEEE Transactions on Neural Networks.

[27]  J.J. Fuchs,et al.  Convergence of a Sparse Representations Algorithm Applicable to Real or Complex Data , 2007, IEEE Journal of Selected Topics in Signal Processing.

[28]  Zhaoshui He,et al.  Improved FOCUSS Method With Conjugate Gradient Iterations , 2009, IEEE Transactions on Signal Processing.

[29]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[30]  Jian Yang,et al.  Sparse Approximation to the Eigensubspace for Discrimination , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[32]  Martin J. McKeown,et al.  Underdetermined Anechoic Blind Source Separation via $\ell^{q}$-Basis-Pursuit With $q≪1$ , 2007, IEEE Transactions on Signal Processing.

[33]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[34]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[35]  Davies Rémi Gribonval.  Restricted Isometry Constants Where Lp Sparse Recovery Can Fail for 0 , 2008 .

[36]  K. Kreutz-Delgado,et al.  Deriving algorithms for computing sparse solutions to linear inverse problems , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[37]  Yong Xiang,et al.  Time-Frequency Approach to Underdetermined Blind Source Separation , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Pierre Vandergheynst,et al.  On the exponential convergence of matching pursuits in quasi-incoherent dictionaries , 2006, IEEE Transactions on Information Theory.

[39]  Huaguang Zhang,et al.  Global Asymptotic Stability and Robust Stability of a Class of Cohen-Grossberg Neural Networks With Mixed Delays , 2009, IEEE Trans. Circuits Syst. I Regul. Pap..

[40]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[41]  Rémi Gribonval,et al.  Restricted Isometry Constants Where $\ell ^{p}$ Sparse Recovery Can Fail for $0≪ p \leq 1$ , 2009, IEEE Transactions on Information Theory.

[42]  Rahul Garg,et al.  Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property , 2009, ICML '09.

[43]  Mark D. Plumbley.  On Polar Polytopes and the Recovery of Sparse Representations , 2007, IEEE Trans. Inf. Theory.

[44]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[45]  Yuanqing Li,et al.  Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation , 2008, IEEE Transactions on Neural Networks.

[46]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[47]  Paul Tseng.  Further Results on Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise , 2009, IEEE Transactions on Information Theory.

[48]  Sundeep Rangan,et al.  Orthogonal Matching Pursuit From Noisy Random Measurements: A New Analysis , 2009, NIPS.

[49]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[50]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[51]  Yonina C. Eldar,et al.  Blind Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[52]  Bhaskar D. Rao,et al.  Signal processing with the sparseness constraint , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[53]  Kwang Suk Park,et al.  Regularized FOCUSS algorithm for EEG/MEG source imaging , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[54]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[55]  Michael I. Jordan,et al.  On Convergence Properties of the EM Algorithm for Gaussian Mixtures , 1996, Neural Computation.

[56]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[57]  Jie Chen,et al.  Theoretical Results on Sparse Representations of Multiple-Measurement Vectors , 2006, IEEE Transactions on Signal Processing.

[58]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[59]  Jian Yang,et al.  Sparse Representation Classifier Steered Discriminative Projection With Applications to Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[60]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[61]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[62]  Mineichi Kudo,et al.  Performance analysis of minimum /spl lscr//sub 1/-norm solutions for underdetermined source separation , 2004, IEEE Transactions on Signal Processing.

[63]  Bhaskar D. Rao,et al.  Subset selection in noise based on diversity measure minimization , 2003, IEEE Trans. Signal Process..

[64]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..