Rate of Convergence of the FOCUSS Algorithm

Focal underdetermined system solver (FOCUSS) is a powerful method for basis selection and sparse representation, where it employs the <inline-formula> <tex-math notation="LaTeX">$\ell _{\vphantom {R_{j_{l}}}p}$ </tex-math></inline-formula>-norm with <inline-formula> <tex-math notation="LaTeX">$p\in (0,2)$ </tex-math></inline-formula> to measure the sparsity of solutions. In this paper, we give a systematical analysis on the rate of convergence of the FOCUSS algorithm with respect to <inline-formula> <tex-math notation="LaTeX">$p\in (0,2)$ </tex-math></inline-formula>. We prove that the FOCUSS algorithm converges superlinearly for <inline-formula> <tex-math notation="LaTeX">$0<p<1$ </tex-math></inline-formula> and linearly for <inline-formula> <tex-math notation="LaTeX">$1\le p<2$ </tex-math></inline-formula> usually, but may superlinearly in some very special scenarios. In addition, we verify its rates of convergence with respect to <inline-formula> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> by numerical experiments.

[1]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[2]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[3]  Michael I. Jordan,et al.  On Convergence Properties of the EM Algorithm for Gaussian Mixtures , 1996, Neural Computation.

[4]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[5]  K. Kreutz-Delgado,et al.  Deriving algorithms for computing sparse solutions to linear inverse problems , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[6]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[7]  Bhaskar D. Rao,et al.  Signal processing with the sparseness constraint , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[8]  Bhaskar D. Rao,et al.  An affine scaling methodology for best basis selection , 1999, IEEE Trans. Signal Process..

[9]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[10]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[11]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[13]  Bhaskar D. Rao,et al.  Subset selection in noise based on diversity measure minimization , 2003, IEEE Trans. Signal Process..

[14]  Kwang Suk Park,et al.  Regularized FOCUSS algorithm for EEG/MEG source imaging , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[15]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[16]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[17]  Martin J. McKeown,et al.  Underdetermined Anechoic Blind Source Separation via $\ell^{q}$-Basis-Pursuit With $q≪1$ , 2007, IEEE Transactions on Signal Processing.

[18]  Jong Chul Ye,et al.  Improved k–t BLAST and k–t SENSE using FOCUSS , 2007, Physics in medicine and biology.

[19]  J. C. Ye,et al.  Projection reconstruction MR imaging using FOCUSS , 2007, Magnetic resonance in medicine.

[20]  J.J. Fuchs,et al.  Convergence of a Sparse Representations Algorithm Applicable to Real or Complex Data , 2007, IEEE Journal of Selected Topics in Signal Processing.

[21]  Yuanqing Li,et al.  Equivalence Probability and Sparsity of Two Sparse Solutions in Sparse Representation , 2008, IEEE Transactions on Neural Networks.

[22]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[23]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Shengli Xie,et al.  Mixing Matrix Estimation From Sparse Mixtures With Unknown Number of Sources , 2011, IEEE Transactions on Neural Networks.

[25]  Yong Xiang,et al.  Nonnegative Blind Source Separation by Sparse Component Analysis Based on Determinant Measure , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Guillaume Obozinski,et al.  Sparse methods for machine learning Theory and algorithms , 2012 .

[27]  Yong Xiang,et al.  Time-Frequency Approach to Underdetermined Blind Source Separation , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Zongben Xu,et al.  $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Jian Yang,et al.  Sparse Approximation to the Eigensubspace for Discrimination , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Ran He,et al.  Two-Stage Nonnegative Sparse Representation for Large-Scale Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[32]  Jian Yang,et al.  Sparse Representation Classifier Steered Discriminative Projection With Applications to Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Wotao Yin,et al.  Improved Iteratively Reweighted Least Squares for Unconstrained Smoothed 퓁q Minimization , 2013, SIAM J. Numer. Anal..

[34]  Zhaoshui He,et al.  Convergence Analysis of the FOCUSS Algorithm , 2015, IEEE Transactions on Neural Networks and Learning Systems.