Non-negative Matrix Factorization with Quasi-Newton Optimization

Non-negative matrix factorization (NMF) is an emerging method with wide spectrum of potential applications in data analysis, feature extraction and blind source separation. Currently, most applications use relative simple multiplicative NMF learning algorithms which were proposed by Lee and Seung, and are based on minimization of the Kullback-Leibler divergence and Frobenius norm. Unfortunately, these algorithms are relatively slow and often need a few thousands of iterations to achieve a local minimum. In order to increase a convergence rate and to improve performance of NMF, we proposed to use a more general cost function: so-called Amari alpha divergence. Taking into account a special structure of the Hessian of this cost function, we derived a relatively simple second-order quasi-Newton method for NMF. The validity and performance of the proposed algorithm has been extensively tested for blind source separation problems, both for signals and images. he performance of the developed NMF algorithm is illustrated for separation of statistically dependent signals and images from their linear mixtures.

[1]  R. Plemmons,et al.  NONNEGATIVE MATRIX FACTORIZATION AND APPLICATIONS , 2005 .

[2]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[3]  Timothy R. C. Read,et al.  Goodness-Of-Fit Statistics for Discrete Multivariate Data , 1988 .

[4]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[5]  Jane Cullum The effective choice of the smoothing norm in regularization , 1979 .

[6]  Per Christian Hansen,et al.  Rank-Deficient and Discrete Ill-Posed Problems , 1996 .

[7]  J. Navarro-Pedreño Numerical Methods for Least Squares Problems , 1996 .

[8]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[9]  Shun-ichi Amari,et al.  Information geometry of the EM and em algorithms for neural networks , 1995, Neural Networks.

[10]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[11]  W. J. Eckert The IBM pluggable sequence relay calculator , 1948 .

[12]  Michael W. Berry,et al.  Text Mining Using Non-Negative Matrix Factorizations , 2004, SDM.

[13]  P. Hansen Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[14]  Inderjit S. Dhillon,et al.  Generalized Nonnegative Matrix Approximations with Bregman Divergences , 2005, NIPS.

[15]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[16]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[17]  Charles L. Byrne,et al.  Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative methods , 1998, IEEE Trans. Image Process..