Seeking an appropriate alternative least squares algorithm for nonnegative tensor factorizations

Alternative least squares (ALS) algorithm is considered as a “work-horse” algorithm for general tensor factorizations. A common form of this algorithm for nonnegative tensor factorizations (NTF) is always combined with a nonlinear projection (rectifier) to enforce nonnegative entries during the estimation. Such simple modification often provides acceptable results for general data. However, this does not establish an appropriate ALS algorithm for NTF. This kind of ALS algorithm often converges slowly, or cannot converge to the desired solution, especially for collinear data. To this end, in this paper, we reinvestigate the nonnegative quadratic programming, propose a recursive method for solving this problem. Then, we formulate a novel ALS algorithm for NTF. The validity and high performance of the proposed algorithm has been confirmed for difficult benchmarks, and also in an application of object classification, and analysis of EEG signals.

[1]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[2]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[3]  Pierre Comon,et al.  Enhanced Line Search: A Novel Method to Accelerate PARAFAC , 2008, SIAM J. Matrix Anal. Appl..

[4]  Andrzej Cichocki,et al.  Nonnegative matrix factorization with constrained second-order optimization , 2007, Signal Process..

[5]  Nicolas Gillis,et al.  Nonnegative Factorization and The Maximum Edge Biclique Problem , 2008, 0810.4225.

[6]  Ji-Hong Wang,et al.  Discriminant image resolution: a novel multivariate image analysis method utilizing a spatial classification constraint in addition to bilinear nonnegativity , 2005 .

[7]  P. Paatero.  A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[8]  Liqing Zhang,et al.  Noninvasive BCIs: Multiway Signal-Processing Array Decompositions , 2008, Computer.

[9]  Andrzej Cichocki,et al.  Nonnegative matrix factorization with quadratic programming , 2008, Neurocomputing.

[10]  Daniel D. Lee,et al.  Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines , 2002, NIPS.

[11]  C. D. Meyer,et al.  Initializations for the Nonnegative Matrix Factorization , 2006 .

[12]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[13]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[14]  Andrzej Cichocki,et al.  PARAFAC algorithms for large-scale problems , 2011, Neurocomputing.

[15]  P. Paatero.  Least squares formulation of robust non-negative factor analysis , 1997 .

[16]  Andrzej Cichocki,et al.  Nonnegative Tensor Factorization for Continuous EEG Classification , 2007, Int. J. Neural Syst..

[17]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[18]  Manfred K. Warmuth,et al.  Exponentiated Gradient Versus Gradient Descent for Linear Predictors , 1997, Inf. Comput..

[19]  P. Hopke,et al.  Application of modified alternating least squares regression to spectroscopic image analysis , 2003 .

[20]  Lieven De Lathauwer,et al.  An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA , 2008, Signal Process..

[21]  Seungjin Choi,et al.  Nonnegative Tucker Decomposition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Nicholas J. Higham,et al.  Matlab guide, Second Edition , 2005 .

[23]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[24]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[25]  Nikolaos G. Bourbakis,et al.  Neural Network Approach for Image chromatic Adaptation for Skin Color Detection , 2007, Int. J. Neural Syst..

[26]  Andrzej Cichocki,et al.  Non-Negative Tensor Factorization using Alpha and Beta Divergences , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[27]  Daniel D. Lee,et al.  Multiplicative Updates for Nonnegative Quadratic Programming , 2007, Neural Computation.

[28]  Feng Qianjin,et al.  Projected gradient methods for Non-negative Matrix Factorization based relevance feedback algorithm in medical image retrieval , 2011 .

[29]  L. Lathauwer,et al.  An enhanced plane search scheme for complex-valued tensor decompositions , 2010 .

[30]  Lars Kai Hansen,et al.  ERPWAVELAB A toolbox for multi-channel analysis of time–frequency transformed event related potentials , 2007, Journal of Neuroscience Methods.

[31]  Lars Kai Hansen,et al.  Decomposing the time-frequency representation of EEG using non-negative matrix and multi-way factorization , 2006 .

[32]  Nicholas J. Higham,et al.  Accuracy and stability of numerical algorithms, Second Edition , 2002 .

[33]  Seungjin Choi,et al.  Semi-Supervised Nonnegative Matrix Factorization , 2010, IEEE Signal Processing Letters.

[34]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[35]  Andrzej Cichocki,et al.  Multilayer Nonnegative Matrix Factorization Using Projected Gradient Approaches , 2007, Int. J. Neural Syst..