A non-monotonic method for large-scale non-negative least squares

We present a new algorithm for solving the non-negative least-squares (NNLS) problem. Our algorithm extends the unconstrained quadratic optimization algorithm of Barzilai and Borwein (BB) [J. Barzilai and J. M. Borwein; Two-Point Step Size Gradient Methods. IMA J. Numer. Anal. 1988.] to handle nonnegativity constraints. Our extension differs from other constrained BB variants in simple but crucial aspects, the most notable being our modification to the BB stepsize itself. Our stepsize computation takes into account the nonnegativity constraints, and is further refined by a stepsize scaling strategy. These changes, in combination with orthogonal projections onto the nonnegative orthant, yield an effective NNLS algorithm. We compare our algorithm with several competing approaches, including established bound-constrained solvers, popular BB-based methods, and also a specialised NNLS algorithm. On several synthetic and real-world datasets our method displays highly competitive empirical performance.

[1]  J. Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[2]  R. Bro,et al.  A fast non‐negativity‐constrained least squares algorithm , 1997 .

[3]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[4]  Klaus Schittkowski.  The Numerical Solution of Constrained Linear Least-squares Problems , 1983 .

[5]  Roger Fletcher,et al.  Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming , 2005, Numerische Mathematik.

[6]  Philip E. Gill,et al.  Practical optimization , 1981 .

[7]  J. M. Martínez,et al.  SPG: Software for Convex-Constrained Optimization , 2008 .

[8]  José Mario Martínez,et al.  Large-Scale Active-Set Box-Constrained Optimization Method with Spectral Projected Gradients , 2002, Comput. Optim. Appl..

[9]  Arkadi Nemirovski,et al.  Non-euclidean restricted memory level method for large-scale convex optimization , 2005, Math. Program..

[10]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[11]  Gerardo Toraldo,et al.  On the Solution of Large Quadratic Programming Problems with Bound Constraints , 1991, SIAM J. Optim..

[12]  Luigi Grippo,et al.  Nonmonotone Globalization Techniques for the Barzilai-Borwein Gradient Method , 2002, Comput. Optim. Appl..

[13]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[14]  J. B. Rosen.  The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[15]  William W. Hager,et al.  A New Active Set Algorithm for Box Constrained Optimization , 2006, SIAM J. Optim..

[16]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[17]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .

[18]  李幼升,et al.  Ph , 1989 .

[19]  Subhash Challa,et al.  Nonnegative-Least-Square Classifier for Face Recognition , 2009, ISNN.

[20]  Inderjit S. Dhillon,et al.  Fast Newton-type Methods for the Least Squares Nonnegative Matrix Approximation Problem , 2007, SDM.

[21]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[22]  F. Ana,et al.  A new method for large-scale box constrained convex quadratic minimization problems , 1995 .

[23]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[24]  Bernhard Schölkopf,et al.  Efficient filter flow for space-variant multiframe blind deconvolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  M. Bierlaire,et al.  On iterative algorithms for linear least squares problems with bound constraints , 1991 .

[26]  H. Zaidi,et al.  Advances in PET Image Reconstruction. , 2007, PET clinics.

[27]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[28]  Marcos Raydan,et al.  Nonmonotone Spectral Methods for Large-Scale Nonlinear Systems , 2003, Optim. Methods Softw..

[29]  J. Nagy,et al.  Enforcing nonnegativity in image reconstruction algorithms , 2000, SPIE Optics + Photonics.

[30]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[32]  J. J. Moré,et al.  On the identification of active constraints , 1988 .

[33]  J. M. Martínez,et al.  Inexact spectral projected gradient methods on convex sets , 2003 .

[34]  Chih-Jen Lin,et al.  Newton's Method for Large Bound-Constrained Optimization Problems , 1999, SIAM J. Optim..

[35]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[36]  José Mario Martínez,et al.  Algorithm 813: SPG—Software for Convex-Constrained Optimization , 2001, TOMS.

[37]  M. Raydan.  On the Barzilai and Borwein choice of steplength for the gradient method , 1993 .

[38]  J. B. Rosen.  The gradient projection method for nonlinear programming: Part II , 1961 .