ADMM penalty parameter selection with krylov subspace recycling technique for sparse coding

Sparse representations are widely used in a broad variety of fields. A number of different methods have been proposed to solve the sparse coding problem, of which the alternating direction method of multipliers (ADMM) is one of the most popular. One of the disadvantages of this method, however, is the need to select an algorithm parameter, the penalty parameter, that has a significant effect on the rate of convergence of the algorithm. Although a number of heuristic methods have been proposed, as yet there is no general theory providing a good choice of this parameter for all problems. One obvious approach would be to try a number of different parameters at each iteration, proceeding further with the one that delivers the best reduction in functional value, but this would involve a substantial increase in computational cost. We show that, when solving the sparse coding problem for a dictionary corresponding to an operator with a fast transform, requiring iterative methods to solve the main linear system arising in the ADMM solution, it is possible to explore a large range of parameters at marginal additional cost, thus greatly improving the robustness of the method to the choice of penalty parameter.

[1]  Åke Björck,et al.  Numerical methods for least square problems , 1996 .

[2]  Guillermo Sapiro,et al.  Sparse Representation for Computer Vision and Pattern Recognition , 2010, Proceedings of the IEEE.

[3]  Michael Elad,et al.  Image Denoising with Shrinkage and Redundant Representations , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[4]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[5]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[6]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[7]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[8]  Behnam Jafarpour,et al.  Effective solution of nonlinear subsurface flow inverse problems in sparse bases , 2010 .

[9]  Stefano Di Cairano,et al.  Alternating direction method of multipliers for strictly convex quadratic programs: Optimal parameter selection , 2014, 2014 American Control Conference.

[10]  Brendt Wohlberg,et al.  Efficient Algorithms for Convolutional Sparse Representations , 2016, IEEE Transactions on Image Processing.

[11]  Zheng Xu,et al.  Adaptive ADMM with Spectral Penalty Parameter Selection , 2016, AISTATS.

[12]  Jean Ponce,et al.  Sparse Modeling for Image and Vision Processing , 2014, Found. Trends Comput. Graph. Vis..

[13]  Brendt Wohlberg,et al.  ADMM Penalty Parameter Selection by Residual Balancing , 2017, ArXiv.

[14]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[15]  David Zhang,et al.  A Survey of Sparse Representation: Algorithms and Applications , 2015, IEEE Access.

[16]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[17]  Brendt Wohlberg,et al.  Performance comparison of iterative reweighting methods for total variation regularization , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[18]  Mac McKee,et al.  Applicability of statistical learning algorithms in groundwater quality modeling , 2005 .