First-order Methods for Geodesically Convex Optimization

Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for several first-order algorithms on Hadamard manifolds. Specifically, we prove upper bounds for the global complexity of deterministic and stochastic (sub)gradient methods for optimizing smooth and nonsmooth g-convex functions, both with and without strong g-convexity. Our analysis also reveals how the manifold geometry, especially \emph{sectional curvature}, impacts convergence rates. To the best of our knowledge, our work is the first to provide global complexity analysis for first-order algorithms for general g-convex optimization.

[1]  R. Bishop,et al.  Manifolds of negative curvature , 1969 .

[2]  Yu. D. Burago,et al.  A.D. Alexandrov spaces with curvature bounded below , 1992 .

[3]  C. Udriste,et al.  Convex Functions and Optimization Methods on Riemannian Manifolds , 1994 .

[4]  S. Cowin,et al.  Averaging Anisotropic Elastic Constant Data , 1997 .

[5]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[6]  M. Bridson,et al.  Metric Spaces of Non-Positive Curvature , 1999 .

[7]  Louis J. Billera,et al.  Geometry of the Space of Phylogenetic Trees , 2001, Adv. Appl. Math..

[8]  R. McCann,et al.  A Riemannian interpolation inequality à la Borell, Brascamp and Lieb , 2001 .

[9]  D. Burago,et al.  A Course in Metric Geometry , 2001 .

[10]  Maher Moakher,et al.  Means and Averaging in the Group of Rotations , 2002, SIAM J. Matrix Anal. Appl..

[11]  Max‐K. von Renesse.  Heat Kernel Comparison on Alexandrov Spaces with Curvature Bounded Below , 2004 .

[12]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[13]  Maher Moakher,et al.  A Differential Geometric Approach to the Geometric Mean of Symmetric Positive-Definite Matrices , 2005, SIAM J. Matrix Anal. Appl..

[14]  Stephen P. Boyd,et al.  A tutorial on geometric programming , 2007, Optimization and Engineering.

[15]  P. Thomas Fletcher,et al.  Riemannian geometry for the statistical analysis of diffusion tensor data , 2007, Signal Process..

[16]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[17]  Hao Shen,et al.  Fast Kernel-Based Independent Component Analysis , 2009, IEEE Transactions on Signal Processing.

[18]  R. Bishop,et al.  Manifolds of negative curvature , 1969 .

[19]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[20]  Bas Lemmens,et al.  Nonlinear Perron-Frobenius Theory , 2012 .

[21]  Ami Wiesel,et al.  Geodesic Convexity and Covariance Estimation , 2012, IEEE Transactions on Signal Processing.

[22]  Brian C. Lovell,et al.  Sparse Coding and Dictionary Learning for Symmetric Positive Definite Matrices: A Kernel Approach , 2012, ECCV.

[23]  Mark W. Schmidt,et al.  A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method , 2012, ArXiv.

[24]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[25]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[26]  Dario Bini,et al.  Computing the Karcher mean of symmetric positive definite matrices , 2013 .

[27]  Bamdev Mishra,et al.  Low-Rank Optimization with Trace Norm Penalty , 2011, SIAM J. Optim..

[28]  Ami Wiesel,et al.  Multivariate Generalized Gaussian Distribution: Convexity and Graphical Models , 2013, IEEE Transactions on Signal Processing.

[29]  M. Bacák.  Convex Analysis and Optimization in Hadamard Spaces , 2014 .

[30]  Xin-Guo Liu,et al.  Maximization of Matrix Trace Function of Product Stiefel Manifolds , 2015, SIAM J. Matrix Anal. Appl..

[31]  Suvrit Sra,et al.  Conic Geometric Optimization on the Manifold of Positive Definite Matrices , 2013, SIAM J. Optim..

[32]  Suvrit Sra,et al.  Matrix Manifold Optimization for Gaussian Mixtures , 2015, NIPS.

[33]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere II: Recovery by Riemannian Trust-Region Method , 2015, IEEE Transactions on Information Theory.