Extended SMART Algorithms for Non-negative Matrix Factorization

In this paper we derive a family of new extended SMART (Simultaneous Multiplicative Algebraic Reconstruction Technique) algorithms for Non-negative Matrix Factorization (NMF). The proposed algorithms are characterized by improved efficiency and convergence rate and can be applied for various distributions of data and additive noise. Information theory and information geometry play key roles in the derivation of new algorithms. We discuss several loss functions used in information theory which allow us to obtain generalized forms of multiplicative NMF learning adaptive algorithms. We also provide flexible and relaxed forms of the NMF algorithms to increase convergence speed and impose an additional constraint of sparsity. The scope of these results is vast since discussed generalized divergence functions include a large number of useful loss functions such as the Amari α– divergence, Relative entropy, Bose-Einstein divergence, Jensen-Shannon divergence, J-divergence, Arithmetic-Geometric (AG) Taneja divergence, etc. We applied the developed algorithms successfully to Blind (or semi blind) Source Separation (BSS) where sources may be generally statistically dependent, however are subject to additional constraints such as nonnegativity and sparsity. Moreover, we applied a novel multilayer NMF strategy which improves performance of the most proposed algorithms.

[1]  Lucas C. Parra,et al.  Recovery of constituent spectra using non-negative matrix factorization , 2003, SPIE Optics + Photonics.

[2]  Mihoko Minami,et al.  Robust Blind Source Separation by Beta Divergence , 2002, Neural Computation.

[3]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[4]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[5]  Allan Kardec Barros,et al.  Independent Component Analysis and Blind Source Separation , 2007, Signal Processing.

[6]  Shun-ichi Amari,et al.  Information geometry of the EM and em algorithms for neural networks , 1995, Neural Networks.

[7]  Seungjin Choi,et al.  Nonnegative features of spectro-temporal sounds for classification , 2005, Pattern Recognit. Lett..

[8]  Richard J. Mammone,et al.  Use of non-negative matrix factorization for language model adaptation in a lecture transcription task , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[9]  Charles L. Byrne,et al.  Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative methods , 1998, IEEE Trans. Image Process..

[10]  Nicol N. Schraudolph,et al.  Gradient-based manipulation of nonparametric entropy estimates , 2004, IEEE Transactions on Neural Networks.

[11]  Stan Z. Li,et al.  Local non-negative matrix factorization as a visual representation , 2002, Proceedings 2nd International Conference on Development and Learning. ICDL 2002.

[12]  Bernt Schiele,et al.  Introducing a weighted non-negative matrix factorization for image classification , 2003, Pattern Recognit. Lett..

[13]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[14]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[15]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[16]  Andrzej Cichocki,et al.  Constrained non-Negative Matrix Factorization Method for EEG Analysis in Early Detection of Alzheimer Disease , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[17]  Mark D. Plumbley,et al.  BLIND SEPARATION OF POSITIVE SOURCES USING NON-NEGATIVE PC A , 2003 .

[18]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[19]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[20]  Jun Zhang,et al.  Divergence Function, Duality, and Convex Analysis , 2004, Neural Computation.

[21]  T. Adalı,et al.  Non-Negative Matrix Factorization with Orthogonality Constraints for Chemical Agent Detection in Raman Spectra , 2005, 2005 IEEE Workshop on Machine Learning for Signal Processing.

[22]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[23]  Timothy R. C. Read,et al.  Goodness-Of-Fit Statistics for Discrete Multivariate Data , 1988 .

[24]  Ananda Sen,et al.  The Theory of Dispersion Models , 1997, Technometrics.

[25]  Yin Zhang,et al.  Interior-Point Gradient Method for Large-Scale Totally Nonnegative Least Squares Problems , 2005 .

[26]  C. Byrne.  Choosing parameters in block-iterative or ordered subset reconstruction algorithms , 2005, IEEE Transactions on Image Processing.

[27]  Anne Lohrli.  Chapman and Hall , 1985 .

[28]  Raul Kompass,et al.  A Generalized Divergence Measure for Nonnegative Matrix Factorization , 2007, Neural Computation.

[29]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[30]  Inderjit S. Dhillon,et al.  Generalized Nonnegative Matrix Approximations with Bregman Divergences , 2005, NIPS.