On the limited memory BFGS method for large scale optimization

We study the numerical performance of a limited memory quasi-Newton method for large scale optimization, which we call the L-BFGS method. We compare its performance with that of the method developed by Buckley and LeNir (1985), which combines cycles of BFGS steps and conjugate direction steps. Our numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence. We show that the L-BFGS method can be greatly accelerated by means of a simple scaling. We then compare the L-BFGS method with the partitioned quasi-Newton method of Griewank and Toint (1982a). The results show that, for some problems, the partitioned quasi-Newton method is clearly superior to the L-BFGS method. However we find that for other problems the L-BFGS method is very competitive due to its low iteration cost. We also study the convergence properties of the L-BFGS method, and prove global convergence on uniformly convex problems.

[1]  J. D. Pearson.  ON VARIABLE METRIC METHODS OF MINIMIZATION , 1968 .

[2]  J. D. Pearson.  Variable metric methods of minimisation , 1969, Comput. J..

[3]  G. R. Walsh,et al.  Methods Of Optimization , 1976 .

[4]  A. Perry.  A Class of Conjugate Gradient Algorithms with a Two-Step Variable Metric Memory , 1977 .

[5]  D. Shanno.  On the Convergence of a New Conjugate Gradient Algorithm , 1978 .

[6]  P. Toint.  Some numerical results using a sparse matrix updating formula in unconstrained optimization , 1978 .

[7]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[8]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[9]  Albert G. Buckley,et al.  A combined conjugate-gradient quasi-Newton minimization algorithm , 1978, Math. Program..

[10]  L. Nazareth.  A Relationship between the BFGS and Conjugate Gradient Algorithms and Its Implications for New Algorithms , 1979 .

[11]  P. Gill,et al.  Conjugate-Gradient Methods for Large-Scale Nonlinear Optimization. , 1979 .

[12]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[13]  David F. Shanno,et al.  Remark on “Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4]” , 1980, TOMS.

[14]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[15]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[16]  Philip E. Gill,et al.  Practical optimization , 1981 .

[17]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[18]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[19]  P. Toint,et al.  Local convergence analysis for partitioned quasi-Newton updates , 1982 .

[20]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[21]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[22]  Albert G. Buckley,et al.  QN-like variable storage conjugate gradients , 1983, Math. Program..

[23]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[24]  Andreas Griewank,et al.  Numerical experiments with partially separable optimization problems , 1984 .

[25]  S. Nash.  Preconditioning of Truncated-Newton Methods , 1985 .

[26]  Albert G. Buckley,et al.  Algorithm 630: BBVSCG–a variable-storage algorithm for function minimization , 1985, TOMS.

[27]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[28]  R. Schnabel,et al.  A view of unconstrained optimization , 1989 .

[29]  Claude Lemaréchal,et al.  Some numerical experiments with variable-storage quasi-Newton algorithms , 1989, Math. Program..

[30]  J. Nocedal,et al.  A tool for the analysis of Quasi-Newton methods with application to unconstrained minimization , 1989 .