Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC

The damped Gauss--Newton (dGN) algorithm for CANDECOMP/PARAFAC (CP) decomposition can handle the challenges of collinearity of factors and different magnitudes of factors; nevertheless, for factorization of an order-$N$ tensor of size $I_1\times\cdots\times I_N$ with rank $R$, the algorithm is computationally demanding due to construction of a large approximate Hessian of size $(RT \times RT)$ and its inversion where $T = \sum_n I_n$. In this paper, we propose a fast implementation of the dGN algorithm which is based on novel expressions of the inverse approximate Hessian in block form. The new implementation has lower computational complexity, besides computation of the gradient (this part is common to both methods), requiring the inversion of a matrix of size $NR^2\times NR^2$, which is much smaller than the whole approximate Hessian, if $T \gg NR$. In addition, the implementation has lower memory requirements, because neither the Hessian nor its inverse never needs to be stored in its entirety. A varia...

[1]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[2]  R. Bro,et al.  PARAFAC and missing values , 2005 .

[3]  H. B. Nielsen.  DAMPING PARAMETER IN MARQUARDT ’ S METHOD , 1999 .

[4]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[5]  P. Paatero.  Least squares formulation of robust non-negative factor analysis , 1997 .

[6]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[7]  Chikio Hayashi,et al.  A NEW ALGORITHM TO SOLVE PARAFAC-MODEL , 1982 .

[8]  G. Saridis.  Parameter estimation: Principles and problems , 1983, Proceedings of the IEEE.

[9]  Nikos D. Sidiropoulos,et al.  Cramer-Rao lower bounds for low-rank decomposition of multidimensional arrays , 2001, IEEE Trans. Signal Process..

[10]  P. Tichavský,et al.  Stability analysis and fast damped-gauss-newton algorithm for INDSCAL tensor decomposition , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[11]  Ben C. Mitchell,et al.  Slowly converging parafac sequences: Swamps and two‐factor degeneracies , 1994 .

[12]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[13]  P. Paatero.  The Multilinear Engine—A Table-Driven, Least Squares Program for Solving Multilinear Problems, Including the n-Way Parallel Factor Analysis Model , 1999 .

[14]  Zbynek Koldovský,et al.  Cramér-Rao-Induced Bounds for CANDECOMP/PARAFAC Tensor Decomposition , 2012, IEEE Transactions on Signal Processing.

[15]  Daniel M. Dunlavy,et al.  A scalable optimization approach for fitting canonical tensor decompositions , 2011 .

[16]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[17]  Rik Pintelon,et al.  A Gauss-Newton-like optimization algorithm for "weighted" nonlinear least-squares problems , 1996, IEEE Trans. Signal Process..

[18]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[19]  Zbynek Koldovský,et al.  Simultaneous search for all modes in multilinear models , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  W. Marsden.  I and J , 2012 .

[21]  Zbynek Koldovský,et al.  Stability of CANDECOMP-PARAFAC tensor decomposition , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Andrzej Cichocki,et al.  Fast damped gauss-newton algorithm for sparse and nonnegative tensor factorization , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Rasmus Bro,et al.  A comparison of algorithms for fitting the PARAFAC model , 2006, Comput. Stat. Data Anal..

[24]  Zbynek Koldovský,et al.  Weight Adjusted Tensor Method for Blind Separation of Underdetermined Mixtures of Nonstationary Sources , 2011, IEEE Transactions on Signal Processing.

[25]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[26]  P. Paatero.  A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[27]  André Uschmajew,et al.  Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation , 2012, SIAM J. Matrix Anal. Appl..

[28]  XIJING GUO,et al.  Uni-mode and Partial Uniqueness Conditions for CANDECOMP/PARAFAC of Three-Way Arrays with Linearly Dependent Loadings , 2012, SIAM J. Matrix Anal. Appl..

[29]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[30]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[31]  A. Cichocki,et al.  Damped Gauss-Newton algorithm for nonnegative Tucker decomposition , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[32]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..