The cyclic Barzilai-–Borwein method for unconstrained optimization

In the cyclic Barzilai–Borwein (CBB) method, the same Barzilai–Borwein (BB) stepsize is reused for m consecutive iterations. It is proved that CBB is locally linearly convergent at a local minimizer with positive definite Hessian. Numerical evidence indicates that when m > n/2 3, where n is the problem dimension, CBB is locally superlinearly convergent. In the special case m = 3 and n = 2, it is proved that the convergence rate is no better than linear, in general. An implementation of the CBB method, called adaptive cyclic Barzilai–Borwein (ACBB), combines a non-monotone line search and an adaptive choice for the cycle length m. In numerical experiments using the CUTEr test problem library, ACBB performs better than the existing BB gradient algorithm, while it is competitive with the well-known PRP+ conjugate gradient algorithm.

[1]  G. Forsythe.  On the asymptotic directions of thes-dimensional optimum gradient method , 1968 .

[2]  L. Grippo,et al.  A nonmonotone line search technique for Newton's method , 1986 .

[3]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[4]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[5]  Marcos Raydan,et al.  Molecular conformations from distance matrices , 1993, J. Comput. Chem..

[6]  M. Raydan.  On the Barzilai and Borwein choice of steplength for the gradient method , 1993 .

[7]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[8]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[9]  Marcos Raydan,et al.  The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem , 1997, SIAM J. Optim..

[10]  Philippe L. Toint,et al.  Non-monotone trust-region algorithms for nonlinear optimization subject to convex constraints , 1997, Math. Program..

[11]  Ernesto G. Birgina,et al.  Automatic differentiation and spectral projected gradient methods for optimal control problems , 1998 .

[12]  J. M. Martínez,et al.  Gradient Method with Retards and Generalizations , 1998 .

[13]  E. Birgin,et al.  Estimation of the Optical Constants and the Thickness of Thin Films Using Unconstrained Optimization , 1999 .

[14]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[15]  Yuhong Dai,et al.  Minimization Algorithms Based on Supervisor and Searcher Cooperation , 2001 .

[16]  José Mario Martínez,et al.  Algorithm 813: SPG—Software for Convex-Constrained Optimization , 2001, TOMS.

[17]  Luigi Grippo,et al.  Nonmonotone Globalization Techniques for the Barzilai-Borwein Gradient Method , 2002, Comput. Optim. Appl..

[18]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[19]  Marcos Raydan,et al.  Relaxed Steepest Descent and Cauchy-Barzilai-Borwein Method , 2002, Comput. Optim. Appl..

[20]  Yuhong Dai.  On the Nonmonotone Line Search , 2002 .

[21]  L. Liao,et al.  R-linear convergence of the Barzilai and Borwein gradient method , 2002 .

[22]  Yuhong Dai.  Alternate step gradient method , 2003 .

[23]  Ya-Xiang Yuan,et al.  Alternate minimization gradient method , 2003 .

[24]  Marcos Raydan,et al.  Nonmonotone Spectral Methods for Large-Scale Nonlinear Systems , 2003, Optim. Methods Softw..

[25]  Hongchao Zhang,et al.  Adaptive Two-Point Stepsize Gradient Algorithm , 2001, Numerical Algorithms.

[26]  Roger Fletcher,et al.  Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming , 2005, Numerische Mathematik.

[27]  Roger Fletcher,et al.  On the Barzilai-Borwein Method , 2005 .

[28]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[29]  Luca Zanni,et al.  Gradient projection methods for quadratic programs and applications in training support vector machines , 2005, Optim. Methods Softw..

[30]  Roger Fletcher,et al.  On the asymptotic behaviour of some new gradient methods , 2005, Math. Program..

[31]  Roger Fletcher,et al.  New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds , 2006, Math. Program..

[32]  William W. Hager,et al.  Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent , 2006, TOMS.

[33]  X. Q. Yang,et al.  A New Gradient Method with an Optimal Stepsize Property , 2006, Comput. Optim. Appl..

[34]  William W. Hager,et al.  A New Active Set Algorithm for Box Constrained Optimization , 2006, SIAM J. Optim..