Proximal Splitting Algorithms: Overrelax them all!

Many problems in statistics, machine learning, signal and image processing, or control can be formulated as convex optimization problems [1–6]. In the age of ‘big data’, with the explosion in size and complexity of the data to process, it is increasingly important to be able to solve convex optimization problems, whose solutions live in very high dimensional spaces [2, 7–10]. There is extensive literature about splitting methods for solving convex optimization problems, with applications in various fields [8, 11–17]. They consist of simple, easy to compute, steps that can deal with the terms in the objective function separately. In this paper, we present several splitting methods in the single umbrella of the forward–backward iteration to solve monotone inclusions, appliedwith preconditioning. In addition, we show that, when the smooth term in the objective function is quadratic, convergence is guaranteed with larger values of the relaxation parameter than previously known. By relaxation, we mean the following: let us consider an iterative algorithm of the form z = T (z), for some operator T , which converges to some fixed point and solution z. Then z tends to be closer than z to z, on average. So, we may want, starting at z, to move further in the direction z − z, which improves the estimate. This yields the relaxed iteration: ⌊ z 1 2 ) = T (z) z = z + ρ(z 1 2 ) − z) . (1)

[1]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[2]  Benar Fux Svaiter,et al.  On Weak Convergence of the Douglas-Rachford Method , 2010, SIAM J. Control. Optim..

[3]  Ming Yan,et al.  A New Primal–Dual Algorithm for Minimizing the Sum of Three Functions with a Linear Operator , 2016, J. Sci. Comput..

[4]  L. Briceño-Arias.  Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions , 2012, 1212.5942.

[5]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[6]  Matthew K. Tam,et al.  A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity , 2018, SIAM J. Optim..

[7]  Jian-Feng Cai,et al.  Split Bregman Methods and Frame Based Image Restoration , 2009, Multiscale Model. Simul..

[8]  Valérie R. Wajs,et al.  A variational formulation for frame-based inverse problems , 2007 .

[9]  Patrick L. Combettes,et al.  Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping , 2014 .

[10]  William W. Hager,et al.  Bregman operator splitting with variable stepsize for total variation image reconstruction , 2013, Comput. Optim. Appl..

[11]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[12]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[13]  James G. Scott,et al.  Proximal Algorithms in Statistics and Machine Learning , 2015, ArXiv.

[14]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[15]  Ernest K. Ryu,et al.  Finding the Forward-Douglas–Rachford-Forward Method , 2019, J. Optim. Theory Appl..

[16]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[17]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[18]  Aix-Marseille.  A Note on the Forward-Douglas – Rachford Splitting for Monotone Inclusion and Convex Optimization , 2017 .

[19]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[20]  Stephen P. Boyd,et al.  A Primer on Monotone Operator Methods , 2015 .

[21]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[22]  José M. Bioucas-Dias,et al.  Restoration of Poissonian Images Using Alternating Direction Optimization , 2010, IEEE Transactions on Image Processing.

[23]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[24]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[25]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[26]  Panagiotis Patrinos,et al.  Asymmetric forward–backward–adjoint splitting for solving monotone inclusions involving three operators , 2016, Comput. Optim. Appl..

[27]  Laurent Condat,et al.  Discrete Total Variation: New Definition and Minimization , 2017, SIAM J. Imaging Sci..

[28]  Kristian Bredies,et al.  A Proximal Point Analysis of the Preconditioned Alternating Direction Method of Multipliers , 2017, J. Optim. Theory Appl..

[29]  Peter Richtárik,et al.  A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent , 2019, AISTATS.

[30]  A. Chambolle,et al.  A remark on accelerated block coordinate descent for computing the proximity operators of a sum of convex functions , 2015 .

[31]  Antonin Chambolle,et al.  Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications , 2017, SIAM J. Optim..

[32]  Michael B. Wakin.  Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity (Starck, J.-L., et al; 2010) [Book Reviews] , 2011, IEEE Signal Processing Magazine.

[33]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[34]  Jian-Feng Cai,et al.  Linearized Bregman Iterations for Frame-Based Image Deblurring , 2009, SIAM J. Imaging Sci..

[35]  Giovanni Chierchia,et al.  Proximity Operators of Discrete Information Divergences , 2016, IEEE Transactions on Information Theory.

[36]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[37]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[38]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[39]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[40]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[41]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[42]  Ali Emrouznejad,et al.  Big Data Optimization: Recent Developments and Challenges , 2016 .

[43]  P. L. Combettes.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[44]  Ernö Robert Csetnek,et al.  Recent Developments on Primal–Dual Splitting Methods with Applications to Convex Minimization , 2014 .

[45]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[46]  F. Bach,et al.  Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning) , 2011 .

[47]  Simon Setzer,et al.  Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing , 2011, International Journal of Computer Vision.

[48]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[49]  P. L. Combettes,et al.  A Dykstra-like algorithm for two monotone operators , 2007 .

[50]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[51]  Wotao Yin,et al.  Splitting Methods in Communication, Imaging, Science, and Engineering , 2017 .

[52]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[53]  Damek Davis,et al.  Forward-Backward-Half Forward Algorithm for Solving Monotone Inclusions , 2017, SIAM J. Optim..

[54]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[55]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2013, J. Optim. Theory Appl..

[56]  Radu Ioan Bot,et al.  On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems , 2013, Mathematical Programming.

[57]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[58]  Colin N. Jones,et al.  Operator Splitting Methods in Control , 2016, Found. Trends Syst. Control..

[59]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[60]  Xue-Cheng Tai,et al.  Some Facts About Operator-Splitting and Alternating Direction Methods , 2016 .

[61]  Laurent Condat.  Fast projection onto the simplex and the l1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pmb {l}_\mathbf {1}$$\end{ , 2015, Mathematical Programming.

[62]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[63]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[64]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[65]  I. M. Otivation.  Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems , 2018 .

[66]  Jonathan Eckstein,et al.  Understanding the Convergence of the Alternating Direction Method of Multipliers: Theoretical and Computational Perspectives , 2015 .

[67]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[68]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[69]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[70]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[71]  P. Tseng.  Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[72]  Laurent Condat,et al.  A Convex Approach to Superresolution and Regularization of Lines in Images , 2019, SIAM J. Imaging Sci..

[73]  Ernest K. Ryu.  Uniqueness of DRS as the 2 operator resolvent-splitting and impossibility of 3 operator resolvent-splitting , 2018, Math. Program..

[74]  J. Moreau.  Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[75]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[76]  José M. Bioucas-Dias,et al.  An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems , 2009, IEEE Transactions on Image Processing.

[77]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[78]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[79]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[80]  Xue-Cheng Tai,et al.  Augmented Lagrangian Method, Dual Methods, and Split Bregman Iteration for ROF, Vectorial TV, and High Order Models , 2010, SIAM J. Imaging Sci..

[81]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[82]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[83]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[84]  B. Mercier.  Topics in Finite Element Solution of Elliptic Problems , 1979 .

[85]  Patrick L. Combettes,et al.  Monotone operator theory in convex optimization , 2018, Math. Program..

[86]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[87]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[88]  Shiqian Ma,et al.  Alternating Proximal Gradient Method for Convex Minimization , 2016, J. Sci. Comput..

[89]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[90]  Nelly Pustelnik,et al.  Proximity Operator of a Sum of Functions; Application to Depth Map Estimation , 2017, IEEE Signal Processing Letters.

[91]  Ming Yan,et al.  Self Equivalence of the Alternating Direction Method of Multipliers , 2014, 1407.7400.

[92]  G. Chen.  Forward-backward splitting techniques: theory and applications , 1994 .

[93]  Lieven Vandenberghe,et al.  On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting , 2018, Math. Program..

[94]  Xavier Bresson,et al.  Geometric Applications of the Split Bregman Method: Segmentation and Surface Reconstruction , 2010, J. Sci. Comput..

[95]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[96]  Xiaoqun Zhang,et al.  A primal-dual fixed point algorithm for minimization of the sum of three convex separable functions , 2015, 1512.09235.

[97]  Simon Setzer,et al.  Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage , 2009, SSVM.

[98]  Laurent Condat,et al.  A Direct Algorithm for 1-D Total Variation Denoising , 2013, IEEE Signal Processing Letters.

[99]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[100]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[101]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[102]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[103]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .