Proximal Splitting Algorithms: A Tour of Recent Advances, with New Twists.

Convex optimization problems, whose solutions live in very high dimensional spaces, have become ubiquitous. To solve them, proximal splitting algorithms are particularly adequate: they consist of simple operations, by handling the terms in the objective function separately. In this overview, we present a selection of recent proximal splitting algorithms within a unified framework, which consists in applying splitting methods for monotone inclusions in primal-dual product spaces, with well-chosen metric. This allows us to derive new variants of the algorithms and to revisit existing convergence results, by extending the parameter ranges in several cases. In particular, when the smooth term in the objective function is quadratic, e.g. for least-squares problems, convergence is guaranteed with larger values of the relaxation parameter than previously known. Such larger values are usually beneficial to the convergence speed in practice.

[1]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[2]  William W. Hager,et al.  Bregman operator splitting with variable stepsize for total variation image reconstruction , 2013, Comput. Optim. Appl..

[3]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[4]  Antonin Chambolle,et al.  Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications , 2017, SIAM J. Optim..

[5]  Jonathan Eckstein Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[6]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[7]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[8]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[9]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[10]  Peter Richtárik,et al.  A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent , 2019, AISTATS.

[11]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[12]  Ali Emrouznejad,et al.  Big Data Optimization: Recent Developments and Challenges , 2016 .

[13]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[14]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[15]  J. Moreau Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[16]  P. Tseng Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[17]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[18]  A. Chambolle,et al.  A remark on accelerated block coordinate descent for computing the proximity operators of a sum of convex functions , 2015 .

[19]  G. Chen Forward-backward splitting techniques: theory and applications , 1994 .

[20]  Nikos Komodakis,et al.  Playing with Duality: An overview of recent primal?dual approaches for solving large-scale optimization problems , 2014, IEEE Signal Process. Mag..

[21]  Laurent Condat,et al.  A Convex Approach to Superresolution and Regularization of Lines in Images , 2019, SIAM J. Imaging Sci..

[22]  Giovanni Chierchia,et al.  Proximity Operators of Discrete Information Divergences , 2016, IEEE Transactions on Information Theory.

[23]  Ming Yan,et al.  A New Primal–Dual Algorithm for Minimizing the Sum of Three Functions with a Linear Operator , 2016, J. Sci. Comput..

[24]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[25]  P. L. Combettes Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[26]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[27]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[28]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[29]  Xiaoqun Zhang,et al.  A primal-dual fixed point algorithm for minimization of the sum of three convex separable functions , 2015, 1512.09235.

[30]  Colin N. Jones,et al.  Operator Splitting Methods in Control , 2016, Found. Trends Syst. Control..

[31]  Ernö Robert Csetnek,et al.  Recent Developments on Primal–Dual Splitting Methods with Applications to Convex Minimization , 2014 .

[32]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[33]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[34]  Bingsheng He,et al.  Convergence Study on the Symmetric Version of ADMM with Larger Step Sizes , 2016, SIAM J. Imaging Sci..

[35]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[36]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[37]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[38]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[39]  Patrick L. Combettes,et al.  Monotone operator theory in convex optimization , 2018, Math. Program..

[40]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[41]  Patrick L. Combettes,et al.  Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping , 2014 .

[42]  Pontus Giselsson,et al.  Nonlinear Forward-Backward Splitting with Projection Correction , 2019, SIAM J. Optim..

[43]  Lieven Vandenberghe,et al.  On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting , 2018, Math. Program..

[44]  Hongpeng Sun,et al.  Analysis of Fully Preconditioned Alternating Direction Method of Multipliers with Relaxation in Hilbert Spaces , 2019, J. Optim. Theory Appl..

[45]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[46]  Matthew K. Tam,et al.  A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity , 2018, SIAM J. Optim..

[47]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[48]  Xue-Cheng Tai,et al.  Augmented Lagrangian Method, Dual Methods, and Split Bregman Iteration for ROF, Vectorial TV, and High Order Models , 2010, SIAM J. Imaging Sci..

[49]  Jian-Feng Cai,et al.  Linearized Bregman Iterations for Frame-Based Image Deblurring , 2009, SIAM J. Imaging Sci..

[50]  Laurent Condat,et al.  Discrete Total Variation: New Definition and Minimization , 2017, SIAM J. Imaging Sci..

[51]  Shiqian Ma,et al.  Alternating Proximal Gradient Method for Convex Minimization , 2016, J. Sci. Comput..

[52]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[53]  James G. Scott,et al.  Proximal Algorithms in Statistics and Machine Learning , 2015, ArXiv.

[54]  Ernest K. Ryu,et al.  Finding the Forward-Douglas–Rachford-Forward Method , 2019, J. Optim. Theory Appl..

[55]  Xavier Bresson,et al.  Geometric Applications of the Split Bregman Method: Segmentation and Surface Reconstruction , 2010, J. Sci. Comput..

[56]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[57]  José M. Bioucas-Dias,et al.  An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems , 2009, IEEE Transactions on Image Processing.

[58]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[59]  Ernest K. Ryu Uniqueness of DRS as the 2 operator resolvent-splitting and impossibility of 3 operator resolvent-splitting , 2018, Math. Program..

[60]  D. Gabay Applications of the method of multipliers to variational inequalities , 1983 .

[61]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[62]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[63]  Nelly Pustelnik,et al.  Proximity Operator of a Sum of Functions; Application to Depth Map Estimation , 2017, IEEE Signal Processing Letters.

[64]  Mohamed-Jalal Fadili,et al.  A Generalized Forward-Backward Splitting , 2011, SIAM J. Imaging Sci..

[65]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[66]  José M. Bioucas-Dias,et al.  Restoration of Poissonian Images Using Alternating Direction Optimization , 2010, IEEE Transactions on Image Processing.

[67]  L. Briceño-Arias Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions , 2012, 1212.5942.

[68]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[69]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[70]  Gregory B. Passty Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[71]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[72]  L. Briceño-Arias,et al.  Primal-dual splittings as fixed point iterations in the range of linear operators , 2019, 1910.02329.

[73]  Laurent Condat,et al.  A Direct Algorithm for 1-D Total Variation Denoising , 2013, IEEE Signal Processing Letters.

[74]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[75]  R. Rockafellar Monotone Operators and the Proximal Point Algorithm , 1976 .

[76]  Panagiotis Patrinos,et al.  Asymmetric forward–backward–adjoint splitting for solving monotone inclusions involving three operators , 2016, Comput. Optim. Appl..

[77]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[78]  Simon Setzer,et al.  Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage , 2009, SSVM.

[79]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[80]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[81]  Laurent Condat,et al.  A Fast Projection onto the Simplex and the l 1 Ball , 2015 .

[82]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[83]  Hugo Raguet,et al.  A note on the forward-Douglas–Rachford splitting for monotone inclusion and convex optimization , 2017, Optim. Lett..

[84]  Patrick L. Combettes,et al.  The Douglas-Rachford Algorithm Converges Only Weakly , 2019, SIAM J. Control. Optim..

[85]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[86]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2013, J. Optim. Theory Appl..

[87]  Ming Yan,et al.  Self Equivalence of the Alternating Direction Method of Multipliers , 2014, 1407.7400.

[88]  Mohamed-Jalal Fadili,et al.  Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity, by Jean-Luc Starck, Fionn Murtagh, and Jalal M. Fadili , 2010, J. Electronic Imaging.

[89]  Kristian Bredies,et al.  A Proximal Point Analysis of the Preconditioned Alternating Direction Method of Multipliers , 2017, J. Optim. Theory Appl..

[90]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[91]  Simon Setzer,et al.  Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing , 2011, International Journal of Computer Vision.

[92]  Valérie R. Wajs,et al.  A variational formulation for frame-based inverse problems , 2007 .

[93]  Jian-Feng Cai,et al.  Split Bregman Methods and Frame Based Image Restoration , 2009, Multiscale Model. Simul..

[94]  Guy Pierra,et al.  Decomposition through formalization in a product space , 1984, Math. Program..

[95]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[96]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[97]  Jonathan Eckstein Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[98]  B. Mercier Topics in Finite Element Solution of Elliptic Problems , 1979 .

[99]  P. L. Combettes,et al.  A Dykstra-like algorithm for two monotone operators , 2007 .

[100]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..