Block-Iterative Algorithms for Non-negative Matrix Approximation

In this paper we present new algorithms for non-negative matrix approximation (NMA), commonly known as the NMF problem. Our methods improve upon the well-known methods of Lee & Seung [12] for both the Frobenius norm as well the Kullback-Leibler divergence versions of the problem. For the latter problem, our results are especially interesting because it seems to have witnessed much lesser algorithmic progress as compared to the Frobenius norm NMA problem. Our algorithms are based on a particular block-iterative acceleration technique for EM, which preserves the multiplicative nature of the updates and also ensures monotonicity. Furthermore, our algorithms also naturally apply to the Bregman-divergence NMA algorithms of [6]. Experimentally,we show that our algorithms outperform the traditional Lee/Seung approach most of the time.

[1]  Petros Drineas,et al.  Workshop on Algorithms for Modern Massive Datasets , 2006 .

[2]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[3]  D. Hunter,et al.  Optimization Transfer Using Surrogate Objective Functions , 2000 .

[4]  Chris H. Q. Ding,et al.  Nonnegative Matrix Factorization and Probabilistic Latent Semantic Indexing: Equivalence Chi-Square Statistic, and a Hybrid Method , 2006, AAAI.

[5]  H. Malcolm Hudson,et al.  Accelerated image reconstruction using ordered subsets of projection data , 1994, IEEE Trans. Medical Imaging.

[6]  Inderjit S. Dhillon,et al.  Fast Newton-type Methods for the Least Squares Nonnegative Matrix Approximation Problem , 2007, SDM.

[7]  Tao Li,et al.  On the Equivalence Between Nonnegative Matrix Factorization and Probabilistic Latent Semantic Indexing , .

[8]  Pasi Aalto,et al.  Matrix factorization methods for analysing diffusion battery data , 1991 .

[9]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[10]  R. Plemmons,et al.  NONNEGATIVE MATRIX FACTORIZATION AND APPLICATIONS , 2005 .

[11]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[12]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[13]  Éric Gaussier,et al.  Relation between PLSA and NMF and implications , 2005, SIGIR '05.

[14]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[15]  Chris H. Q. Ding,et al.  On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing , 2008, Comput. Stat. Data Anal..

[16]  H. Zaidi,et al.  Advances in PET Image Reconstruction. , 2007, PET clinics.

[17]  Heinz H. Bauschke,et al.  Legendre functions and the method of random Bregman projections , 1997 .

[18]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[19]  T. L. Markham.  Factorizations of nonnegative matrices , 1972 .

[20]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[21]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  S. Sra.  Nonnegative Matrix Approximation: Algorithms and Applications , 2006 .

[23]  Tamir Hazan,et al.  Multi-way Clustering Using Super-Symmetric Non-negative Tensor Factorization , 2006, ECCV.

[24]  Bernt Schiele,et al.  Introducing a weighted non-negative matrix factorization for image classification , 2003, Pattern Recognit. Lett..

[25]  Inderjit S. Dhillon,et al.  Clustering with Bregman Divergences , 2005, J. Mach. Learn. Res..

[26]  Alvaro R. De Pierro,et al.  A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography , 1995, IEEE Trans. Medical Imaging.

[27]  L. Shepp,et al.  Maximum Likelihood Reconstruction for Emission Tomography , 1983, IEEE Transactions on Medical Imaging.

[28]  Manfred K. Warmuth,et al.  Relative Loss Bounds for On-Line Density Estimation with the Exponential Family of Distributions , 1999, Machine Learning.

[29]  Andrzej Stachurski,et al.  Parallel Optimization: Theory, Algorithms and Applications , 2000, Parallel Distributed Comput. Pract..

[30]  Robert M. Lewitt,et al.  Overview of methods for image reconstruction from projections in emission computed tomography , 2003, Proc. IEEE.

[31]  P. Green.  On Use of the EM Algorithm for Penalized Likelihood Estimation , 1990 .

[32]  Inderjit S. Dhillon,et al.  Generalized Nonnegative Matrix Approximations with Bregman Divergences , 2005, NIPS.

[33]  M. Daube-Witherspoon,et al.  An Iterative Image Space Reconstruction Algorthm Suitable for Volume ECT , 1986, IEEE Transactions on Medical Imaging.

[34]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[35]  Tao Li,et al.  The Relationships Among Various Nonnegative Matrix Factorization Methods for Clustering , 2006, Sixth International Conference on Data Mining (ICDM'06).

[36]  T. Markham,et al.  Factorizations of completely positive matrices , 1971, Mathematical Proceedings of the Cambridge Philosophical Society.