An algorithm framework of sparse minimization for positive definite quadratic forms

Abstract Many well-known machine learning and pattern recognition methods can be seen as special cases of sparse minimization of Positive Definite Quadratic Forms (PDQF). An algorithm framework of sparse minimization is proposed for PDQF. It is theoretically analyzed to converge to global minimum. The computational complexity is analyzed and compared with the state-of-the-art Fast Iterative Shrinkage-Thresholding Algorithm (FISTA). Some well-known machine learning and pattern recognition methods are illustrated to be optimized by the proposed algorithm framework. Illustrative experiments show that Sparse Representation Classification (SRC) and Least Absolute Shrinkage and Selection Operator (LASSO) via the proposed method converges much faster than several classical methods.

[1]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[2]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[3]  John C. G. Boot,et al.  Quadratic programming: algorithms, anomalies, applications. , 1964 .

[4]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[5]  Kaare Brandt Petersen,et al.  Generative and Filtering Approaches for Overcomplete Representations , 2005 .

[6]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[7]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[8]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[9]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[10]  Nojun Kwak,et al.  Principal Component Analysis Based on L1-Norm Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[12]  Liang-Tien Chia,et al.  Sparse Representation With Kernels , 2013, IEEE Transactions on Image Processing.

[13]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[14]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[15]  Kim-Chuan Toh,et al.  A coordinate gradient descent method for ℓ1-regularized convex minimization , 2011, Comput. Optim. Appl..

[16]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[17]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[18]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[19]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[21]  Jing Wang,et al.  2DPCA with L1-norm for simultaneously robust and sparse modelling , 2013, Neural Networks.

[22]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[23]  Volker Roth,et al.  The generalized LASSO , 2004, IEEE Transactions on Neural Networks.

[24]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[25]  J. Fox.  Applied Regression Analysis, Linear Models, and Related Methods , 1997 .

[26]  Haixian Wang,et al.  Robust sparsity-preserved learning with application to image visualization , 2013, Knowledge and Information Systems.

[27]  Zhong Jin,et al.  Kernel sparse representation based classification , 2012, Neurocomputing.

[28]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[29]  Xuesong Lu,et al.  Fisher Discriminant Analysis With L1-Norm , 2014, IEEE Transactions on Cybernetics.