Generalized Nonnegative Matrix Approximations with Bregman Divergences

Nonnegative matrix approximation (NNMA) is a recent technique for dimensionality reduction and data analysis that yields a parts based, sparse nonnegative representation for nonnegative input data. NNMA has found a wide variety of applications, including text analysis, document clustering, face/image recognition, language modeling, speech processing and many others. Despite these numerous applications, the algorithmic development for computing the NNMA factors has been relatively deficient. This paper makes algorithmic progress by modeling and solving (using multiplicative updates) new generalized NNMA problems that minimize Bregman divergences between the input matrix and its low-rank approximation. The multiplicative update formulae in the pioneering work by Lee and Seung [11] arise as a special case of our algorithms. In addition, the paper shows how to use penalty functions for incorporating constraints other than nonnegativity into the problem. Further, some interesting extensions to the use of "link" functions for modeling nonlinear relationships are also discussed.

[1]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[2]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[3]  Y. Censor,et al.  Parallel Optimization: Theory, Algorithms, and Applications , 1997 .

[4]  P. Paatero.  Least squares formulation of robust non-negative factor analysis , 1997 .

[5]  Y. Censor,et al.  Parallel Optimization:theory , 1997 .

[6]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[7]  P. Paatero.  The Multilinear Engine—A Table-Driven, Least Squares Program for Solving Multilinear Problems, Including the n-Way Parallel Factor Analysis Model , 1999 .

[8]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[9]  J. R. Torres-Lapasió,et al.  Resolution of multicomponent peaks by orthogonal projection approach, positive matrix factorization and alternating least squares , 2000 .

[10]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[11]  Jordi Vitrià,et al.  A weighted non-negative matrix factorization for local representations , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Sanjoy Dasgupta,et al.  A Generalization of Principal Components Analysis to the Exponential Family , 2001, NIPS.

[13]  Richard J. Mammone,et al.  Use of non-negative matrix factorization for language model adaptation in a lecture transcription task , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[14]  Jordi Vitrià,et al.  Analyzing non-negative matrix factorization for image classification , 2002, Object recognition supported by user interaction for service robots.

[15]  Jonathan Foote,et al.  Summarizing video using non-negative similarity matrix factorization , 2002, 2002 IEEE Workshop on Multimedia Signal Processing..

[16]  András Lörincz,et al.  Robust hierarchical image representation using non-negative matrix factorization with sparse code shrinkage preprocessing , 2003 .

[17]  Patrik O. Hoyer,et al.  Non-negative sparse coding , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[18]  Mark D. Plumbley.  Conditions for nonnegative independent component analysis , 2002, IEEE Signal Processing Letters.

[19]  Stan Z. Li,et al.  Local non-negative matrix factorization as a visual representation , 2002, Proceedings 2nd International Conference on Development and Learning. ICDL 2002.

[20]  Jordi Vitrià,et al.  Determining a suitable metric when using non-negative matrix factorization , 2002, Object recognition supported by user interaction for service robots.

[21]  Jordi Vitrià,et al.  Non-negative Matrix Factorization for Face Recognition , 2002, CCIA.

[22]  Bernt Schiele,et al.  Introducing a weighted non-negative matrix factorization for image classification , 2003, Pattern Recognit. Lett..

[23]  Mark D. Plumbley.  Algorithms for nonnegative independent component analysis , 2003, IEEE Trans. Neural Networks.

[24]  P. Smaragdis,et al.  Non-negative matrix factorization for polyphonic music transcription , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[25]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[26]  Tommi S. Jaakkola,et al.  Weighted Low-Rank Approximations , 2003, ICML.

[27]  P. Sajda,et al.  RECOVERY OF CONSTITUENT SPECTRA IN 3D CHEMICAL SHIFT IMAGING USING NON-NEGATIVE MATRIX FACTORIZATION , 2003 .

[28]  Mark D. Plumbley,et al.  BLIND SEPARATION OF POSITIVE SOURCES USING NON-NEGATIVE PC A , 2003 .

[29]  P. Hopke,et al.  Comparison of Positive Matrix Factorization and Multilinear Engine for the source apportionment of particulate pollutants , 2003 .

[30]  Patrik O. Hoyer,et al.  Modeling Receptive Fields with Non-Negative Sparse Coding , 2002, Neurocomputing.

[31]  Sven Behnke,et al.  Discovering hierarchical speech features using convolutional non-negative matrix factorization , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[32]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[33]  Inderjit S. Dhillon,et al.  Concept Decompositions for Large Sparse Text Data Using Clustering , 2004, Machine Learning.

[34]  Joel A. Tropp,et al.  Topics in sparse approximation , 2004 .

[35]  Jong-Hoon Ahn,et al.  MULTIPLE NONNEGATIVE-MATRIX FACTORIZATION OF DYNAMIC PET IMAGES , 2004 .

[36]  Yoram Singer,et al.  Logistic Regression, AdaBoost and Bregman Distances , 2000, Machine Learning.

[37]  Inderjit S. Dhillon,et al.  Minimum Sum-Squared Residue Co-Clustering of Gene Expression Data , 2004, SDM.

[38]  Inderjit S. Dhillon,et al.  Clustering with Bregman Divergences , 2005, J. Mach. Learn. Res..

[39]  Andrzej Cichocki,et al.  Csiszár's Divergences for Non-negative Matrix Factorization: Family of New Algorithms , 2006, ICA.