Non-negative matrix factorization with alpha-divergence

Non-negative matrix factorization (NMF) is a popular technique for pattern recognition, data analysis, and dimensionality reduction, the goal of which is to decompose non-negative data matrix X into a product of basis matrix A and encoding variable matrix S with both A and S allowed to have only non-negative elements. In this paper, we consider Amari's @a-divergence as a discrepancy measure and rigorously derive a multiplicative updating algorithm (proposed in our recent work) which iteratively minimizes the @a-divergence between X and AS. We analyze and prove the monotonic convergence of the algorithm using auxiliary functions. In addition, we show that the same algorithm can be also derived using Karush-Kuhn-Tucker (KKT) conditions as well as the projected gradient. We provide two empirical study for image denoising and EEG classification, showing the interesting and useful behavior of the algorithm in cases where different values of @a (@a=0.5,1,2) are used.

[1]  S. Amari.  Integration of Stochastic Models by Minimizing -Divergence , 2007, Neural Computation.

[2]  Minje Kim,et al.  Monaural Music Source Separation: Nonnegativity, Sparseness, and Shift-Invariance , 2006, ICA.

[3]  Jong-Hoon Ahn,et al.  MULTIPLE NONNEGATIVE-MATRIX FACTORIZATION OF DYNAMIC PET IMAGES , 2004 .

[4]  Pablo Tamayo,et al.  Metagenes and molecular pattern discovery using matrix factorization , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Zhaoshui He,et al.  Extended SMART Algorithms for Non-negative Matrix Factorization , 2006, ICAISC.

[6]  Andrzej Cichocki,et al.  Nonnegative Matrix Factorization for Motor Imagery EEG Classification , 2006, ICANN.

[7]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[8]  Andrzej Cichocki,et al.  Non-negative Matrix Factorization with Quasi-Newton Optimization , 2006, ICAISC.

[9]  Jr. G. Forney,et al.  The viterbi algorithm , 1973 .

[10]  Seungjin Choi,et al.  Nonnegative features of spectro-temporal sounds for classification , 2005, Pattern Recognit. Lett..

[11]  Steven Lemm,et al.  BCI competition 2003-data set III: probabilistic modeling of sensorimotor /spl mu/ rhythms for classification of imaginary hand movements , 2004, IEEE Transactions on Biomedical Engineering.

[12]  Michael W. Berry,et al.  Document clustering using nonnegative matrix factorization , 2006, Inf. Process. Manag..

[13]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[14]  Andrzej Cichocki,et al.  New Algorithms for Non-Negative Matrix Factorization in Applications to Blind Source Separation , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[15]  Daniel D. Lee,et al.  APPLICATION OF NON-NEGATIVE MATRIX FACTORIZATION TO DYNAMIC POSITRON EMISSION TOMOGRAPHY , 2001 .

[16]  Stan Z. Li,et al.  Learning spatially localized, parts-based representation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[17]  Huaiyu Zhu,et al.  Information geometric measurements of generalisation , 1995 .

[18]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[19]  Jdel.R. Millan,et al.  On the need for on-line learning in brain-computer interfaces , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[20]  Chih-Jen Lin,et al.  On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization , 2007, IEEE Transactions on Neural Networks.

[21]  S. M. Ali,et al.  A General Class of Coefficients of Divergence of One Distribution from Another , 1966 .

[22]  Raul Kompass,et al.  A Generalized Divergence Measure for Nonnegative Matrix Factorization , 2007, Neural Computation.

[23]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[24]  Inderjit S. Dhillon,et al.  Generalized Nonnegative Matrix Approximations with Bregman Divergences , 2005, NIPS.

[25]  P. Smaragdis,et al.  Non-negative matrix factorization for polyphonic music transcription , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[26]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[27]  P. Paatero.  Least squares formulation of robust non-negative factor analysis , 1997 .

[28]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[29]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.

[30]  Klaus-Robert Müller,et al.  The BCI competition 2003: progress and perspectives in detection and discrimination of EEG single trials , 2004, IEEE Transactions on Biomedical Engineering.