Distributed Proximal Splitting Algorithms with Rates and Acceleration

We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization. We derive sublinear and linear convergence results with new rates on the function value suboptimality or distance to the solution, as well as new accelerated versions, using varying stepsizes. In addition, we propose distributed variants of these algorithms, which can be accelerated as well. While most existing results are ergodic, our nonergodic results significantly broaden our understanding of primal–dual optimization algorithms.

[1]  Laurent Condat,et al.  From Local SGD to Local Fixed Point Methods for Federated Learning , 2020, ICML.

[2]  Ming Yan,et al.  A New Primal–Dual Algorithm for Minimizing the Sum of Three Functions with a Linear Operator , 2016, J. Sci. Comput..

[3]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[4]  Stephen P. Boyd,et al.  Trend Filtering ∗ , 2009 .

[5]  Colin N. Jones,et al.  Operator Splitting Methods in Control , 2016, Found. Trends Syst. Control..

[6]  Michael Möller,et al.  Collaborative Total Variation: A General Framework for Vectorial TV Models , 2015, SIAM J. Imaging Sci..

[7]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[8]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[9]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[10]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[11]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[12]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[13]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[14]  James G. Scott,et al.  Proximal Algorithms in Statistics and Machine Learning , 2015, ArXiv.

[15]  Laurent Massoulié,et al.  Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks , 2017, ICML.

[16]  Laurent Condat,et al.  An Optimal Algorithm for Strongly Convex Minimization under Affine Constraints , 2021 .

[17]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[18]  Ernö Robert Csetnek,et al.  Recent Developments on Primal–Dual Splitting Methods with Applications to Convex Minimization , 2014 .

[19]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[20]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[21]  F. Bach,et al.  Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning) , 2011 .

[22]  Panagiotis Patrinos,et al.  A New Randomized Block-Coordinate Primal-Dual Proximal Algorithm for Distributed Optimization , 2017, IEEE Transactions on Automatic Control.

[23]  Wotao Yin,et al.  Splitting Methods in Communication, Imaging, Science, and Engineering , 2017 .

[24]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[25]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2013, J. Optim. Theory Appl..

[26]  Convex Optimization in Signal Processing and Communications , 2010 .

[27]  Nikos Komodakis,et al.  Playing with Duality: An overview of recent primal?dual approaches for solving large-scale optimization problems , 2014, IEEE Signal Process. Mag..

[28]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[29]  Laurent Condat,et al.  Discrete Total Variation: New Definition and Minimization , 2017, SIAM J. Imaging Sci..

[30]  Peter Richtárik,et al.  A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent , 2019, AISTATS.

[31]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[32]  Laurent Condat,et al.  Proximal splitting algorithms: Relax them all! , 2019 .

[33]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[34]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[35]  Marios M. Polycarpou,et al.  Sensor Fault Diagnosis , 2016, Found. Trends Syst. Control..

[36]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[37]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[38]  Ali H. Sayed,et al.  Decentralized Proximal Gradient Algorithms With Linear Convergence Rates , 2019, IEEE Transactions on Automatic Control.

[39]  Laurent Condat,et al.  Dualize, Split, Randomize: Fast Nonsmooth Optimization Algorithms , 2020, ArXiv.

[40]  Alexander J. Smola,et al.  Trend Filtering on Graphs , 2014, J. Mach. Learn. Res..

[41]  Ernest K. Ryu,et al.  Proximal-Proximal-Gradient Method , 2017, Journal of Computational Mathematics.

[42]  Laurent Condat,et al.  A Convex Lifting Approach to Image Phase Unwrapping , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[43]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[44]  D. Cremers.  Convex Relaxation Techniques for Segmentation , Stereo and Multiview Reconstruction , 2010 .

[45]  Laurent Condat,et al.  Proximal Splitting Algorithms: A Tour of Recent Advances, with New Twists. , 2020 .

[46]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[47]  Lieven Vandenberghe,et al.  On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting , 2018, Math. Program..

[48]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[49]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[50]  I. M. Otivation.  Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems , 2018 .

[51]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[52]  Laurent Condat,et al.  A Generic Proximal Algorithm for Convex Optimization—Application to Total Variation Minimization , 2014, IEEE Signal Processing Letters.

[53]  Laurent Condat,et al.  A Convex Approach to K-Means Clustering and Image Segmentation , 2017, EMMCVPR.