Nonnegative matrix factorization with constrained second-order optimization

Nonnegative matrix factorization (NMF) solves the following problem: find nonnegative matrices A@?R"+^M^x^R and X@?R"+^R^x^T such that Y@?AX, given only Y@?R^M^x^T and the assigned index R. This method has found a wide spectrum of applications in signal and image processing, such as blind source separation (BSS), spectra recovering, pattern recognition, segmentation or clustering. Such a factorization is usually performed with an alternating gradient descent technique that is applied to the squared Euclidean distance or Kullback-Leibler divergence. This approach has been used in the widely known Lee-Seung NMF algorithms that belong to a class of multiplicative iterative algorithms. It is well known that these algorithms, in spite of their low complexity, are slowly convergent, give only a strictly positive solution, and can easily fall into local minima of a nonconvex cost function. In this paper, we propose to take advantage of the second-order terms of a cost function to overcome the disadvantages of gradient (multiplicative) algorithms. First, a projected quasi-Newton method is presented, where a regularized Hessian with the Levenberg-Marquardt approach is inverted with the Q-less QR decomposition. Since the matrices A and/or X are usually sparse, a more sophisticated hybrid approach based on the gradient projection conjugate gradient (GPCG) algorithm, which was invented by More and Toraldo, is adapted for NMF. The gradient projection (GP) method is exploited to find zero-value components (active), and then the Newton steps are taken only to compute positive components (inactive) with the conjugate gradient (CG) method. As a cost function, we used the @a-divergence that unifies many well-known cost functions. We applied our new NMF method to a BSS problem with mixed signals and images. The results demonstrate the high robustness of our method.

[1]  Lucas C. Parra,et al.  Nonnegative matrix factorization for rapid recovery of constituent spectra in magnetic resonance chemical shift imaging of the brain , 2004, IEEE Transactions on Medical Imaging.

[2]  Yin Zhang,et al.  Interior-Point Gradient Method for Large-Scale Totally Nonnegative Least Squares Problems , 2005 .

[3]  Bernt Schiele,et al.  Introducing a weighted non-negative matrix factorization for image classification , 2003, Pattern Recognit. Lett..

[4]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[5]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[6]  Dietrich Lehmann,et al.  Nonsmooth nonnegative matrix factorization (nsNMF) , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Andrzej Cichocki,et al.  Non-negative Matrix Factorization with Quasi-Newton Optimization , 2006, ICAISC.

[8]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[9]  Andrzej Cichocki,et al.  Regularized Alternating Least Squares Algorithms for Non-negative Matrix/Tensor Factorization , 2007, ISNN.

[10]  Inderjit S. Dhillon,et al.  Concept Decompositions for Large Sparse Text Data Using Clustering , 2004, Machine Learning.

[11]  Andrzej Cichocki,et al.  Multilayer Nonnegative Matrix Factorization Using Projected Gradient Approaches , 2007, Int. J. Neural Syst..

[12]  Lucas C. Parra,et al.  Recovery of constituent spectra using non-negative matrix factorization , 2003, SPIE Optics + Photonics.

[13]  Michael W. Berry,et al.  Document clustering using nonnegative matrix factorization , 2006, Inf. Process. Manag..

[14]  K. Lange,et al.  EM reconstruction algorithms for emission and transmission tomography. , 1984, Journal of computer assisted tomography.

[15]  T. Adalı,et al.  Non-Negative Matrix Factorization with Orthogonality Constraints for Chemical Agent Detection in Raman Spectra , 2005, 2005 IEEE Workshop on Machine Learning for Signal Processing.

[16]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[17]  Timothy R. C. Read,et al.  Goodness-Of-Fit Statistics for Discrete Multivariate Data , 1988 .

[18]  Simon J. Shepherd,et al.  Extracting characteristic patterns from genome-wide expression data by non-negative matrix factorization , 2004, Proceedings. 2004 IEEE Computational Systems Bioinformatics Conference, 2004. CSB 2004..

[19]  Jordi Vitrià,et al.  Analyzing non-negative matrix factorization for image classification , 2002, Object recognition supported by user interaction for service robots.

[20]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[21]  Nanning Zheng,et al.  Non-negative matrix factorization based methods for object recognition , 2004, Pattern Recognit. Lett..

[22]  L. Lucy An iterative technique for the rectification of observed distributions , 1974 .

[23]  Johnathan M. Bardsley,et al.  A Nonnegatively Constrained Convex Programming Method for Image Reconstruction , 2003, SIAM J. Sci. Comput..

[24]  A. Cichocki,et al.  Multilayer nonnegative matrix factorisation , 2006 .

[25]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[26]  Pablo Tamayo,et al.  Metagenes and molecular pattern discovery using matrix factorization , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[27]  Zhaoshui He,et al.  Extended SMART Algorithms for Non-negative Matrix Factorization , 2006, ICAISC.

[28]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[29]  Jong-Hoon Ahn,et al.  MULTIPLE NONNEGATIVE-MATRIX FACTORIZATION OF DYNAMIC PET IMAGES , 2004 .

[30]  Yunde Jia,et al.  Non-negative matrix factorization framework for face recognition , 2005, Int. J. Pattern Recognit. Artif. Intell..

[31]  R. Plemmons,et al.  Optimality, computation, and interpretation of nonnegative matrix factorizations , 2004 .

[32]  Johnathan M. Bardsley A NONNEGATIVELY CONSTRAINED TRUST REGION ALGORITHM FOR THE RESTORATION OF IMAGES WITH AN UNKNOWN BLUR , 2005 .

[33]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[34]  P. Sajda,et al.  RECOVERY OF CONSTITUENT SPECTRA IN 3D CHEMICAL SHIFT IMAGING USING NON-NEGATIVE MATRIX FACTORIZATION , 2003 .

[35]  Oleg Okun,et al.  Fast Nonnegative Matrix Factorization and Its Application for Protein Fold Recognition , 2006, EURASIP J. Adv. Signal Process..

[36]  Gerardo Toraldo,et al.  On the Solution of Large Quadratic Programming Problems with Bound Constraints , 1991, SIAM J. Optim..

[37]  Michael W. Spratling Learning Image Components for Object Recognition , 2006, J. Mach. Learn. Res..

[38]  Seungjin Choi,et al.  Nonnegative features of spectro-temporal sounds for classification , 2005, Pattern Recognit. Lett..

[39]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[40]  Charles L. Byrne,et al.  Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative methods , 1998, IEEE Trans. Image Process..

[41]  A. R. De Pierro,et al.  On the relation between the ISRA and the EM algorithm for positron emission tomography , 1993, IEEE Trans. Medical Imaging.