Fast Newton-type Methods for the Least Squares Nonnegative Matrix Approximation Problem

Nonnegative Matrix Approximation is an effective matrix decomposition technique that has proven to be useful for a wide variety of applications ranging from document analysis and image processing to bioinformatics. There exist a few algorithms for nonnegative matrix approximation (NNMA), for example, Lee & Seung’s multiplicative updates, alternating least squares, and certain gradient descent based procedures. All of these procedures suffer from either slow convergence, numerical instabilities, or at worst, theoretical unsoundness. In this paper we present new and improved algorithms for the least-squares NNMA problem, which are not only theoretically well-founded, but also overcome many of the deficiencies of other methods. In particular, we use non-diagonal gradient scaling to obtain rapid convergence. Our methods provide numerical results superior to both Lee & Seung’s method as well to the alternating least squares (ALS) heuristic, which is known to work well in some situations but has no theoretical guarantees (Berry et al. 2006). Our approach extends naturally to include regularization and box-constraints, without sacrificing convergence guarantees. We present experimental results on both synthetic and realworld datasets to demonstrate the superiority of our methods, in terms of better approximations as well as efficiency.

[1]  C. G. Broyden The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[2]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[3]  D. Shanno Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[4]  D. Goldfarb A family of variable-metric methods derived by variational means , 1970 .

[5]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[6]  D. Bertsekas Projected Newton methods for optimization problems with simple constraints , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[7]  M. Bierlaire,et al.  On iterative algorithms for linear least squares problems with bound constraints , 1991 .

[8]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[9]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[10]  J. Navarro-Pedreño Numerical Methods for Least Squares Problems , 1996 .

[11]  P. Paatero Least squares formulation of robust non-negative factor analysis , 1997 .

[12]  R. Bro,et al.  A fast non‐negativity‐constrained least squares algorithm , 1997 .

[13]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[14]  P. Paatero The Multilinear Engine—A Table-Driven, Least Squares Program for Solving Multilinear Problems, Including the n-Way Parallel Factor Analysis Model , 1999 .

[15]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[16]  Yin Zhang,et al.  Interior-Point Gradient Method for Large-Scale Totally Nonnegative Least Squares Problems , 2005 .

[17]  Yin Zhang,et al.  Accelerating the Lee-Seung Algorithm for Nonnegative Matrix Factorization , 2005 .

[18]  Jing Zhao,et al.  Document Clustering Based on Nonnegative Sparse Matrix Factorization , 2005, ICNC.

[19]  S. Sra Nonnegative Matrix Approximation: Algorithms and Applications , 2006 .

[20]  I. Dhillon,et al.  A New Projected Quasi-Newton Approach for the Nonnegative Least Squares Problem , 2006 .

[21]  Andrzej Cichocki,et al.  Non-negative Matrix Factorization with Quasi-Newton Optimization , 2006, ICAISC.

[22]  Michael W. Berry,et al.  Document clustering using nonnegative matrix factorization , 2006, Inf. Process. Manag..

[23]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[24]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[25]  R. Rosenfeld Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.