Error Preserving Correction: A Method for CP Decomposition at a Target Error Bound

In CANDECOMP/PARAFAC tensor decomposition, degeneracy often occurs in some difficult scenarios, especially, when the rank exceeds the tensor dimension, or when the loading components are highly collinear in several or all modes, or when CPD does not have an optimal solution. In such cases, norms of some rank-1 tensors become significantly large and cancel each other. This makes algorithms getting stuck in local minima while running a huge number of iterations does not improve the decomposition. In this paper, we propose an error preservation correction method to deal with such problem. Our aim is to seek an alternative tensor, which preserves the approximation error, but norms of rank-1 tensor components of the new tensor are minimized. Alternating and all-at-once correction algorithms have been developed for the problem. In addition, we propose a novel CPD with a bound constraint on the norm of the rank-one tensors. The method can be useful for decomposing tensors that cannot be performed by traditional algorithms. Finally, we demonstrate an application of the proposed method in image denoising and decomposition of the weight tensors in convolutional neural networks.

[1]  P. Paatero A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[2]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[3]  Tamara G. Kolda,et al.  A Practical Randomized CP Tensor Decomposition , 2017, SIAM J. Matrix Anal. Appl..

[4]  W. Gander Least squares with a quadratic constraint , 1980 .

[5]  Ben C. Mitchell,et al.  Slowly converging parafac sequences: Swamps and two‐factor degeneracies , 1994 .

[6]  P. Giordani,et al.  Candecomp/Parafac with ridge regularization , 2013 .

[7]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part I: Lemmas for Partitioned Matrices , 2008, SIAM J. Matrix Anal. Appl..

[8]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[9]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[10]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[11]  Alwin Stegeman,et al.  Candecomp/Parafac: From Diverging Components to a Decomposition in Block Terms , 2012, SIAM J. Matrix Anal. Appl..

[12]  J. Kruskal,et al.  A two-stage procedure incorporating good features of both trilinear and quadrilinear models , 1989 .

[13]  Paolo Giordani,et al.  Constrained Candecomp/Parafac via the Lasso , 2013, Psychometrika.

[14]  Andrzej Cichocki,et al.  A further improvement of a fast damped Gauss-Newton algorithm for candecomp-parafac tensor decomposition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[15]  A. Stegeman,et al.  On the Non-Existence of Optimal Solutions and the Occurrence of “Degeneracy” in the CANDECOMP/PARAFAC Model , 2008, Psychometrika.

[16]  Andrew Zisserman,et al.  Speeding up Convolutional Neural Networks with Low Rank Expansions , 2014, BMVC.

[17]  Paul T. Boggs,et al.  Sequential Quadratic Programming , 1995, Acta Numerica.

[18]  Liqing Zhang,et al.  Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Andrzej Cichocki,et al.  Partitioned Alternating Least Squares Technique for Canonical Polyadic Tensor Decomposition , 2016, IEEE Signal Processing Letters.

[20]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[21]  Nico Vervliet,et al.  A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors , 2016, IEEE Journal of Selected Topics in Signal Processing.

[22]  Andrzej Cichocki,et al.  Quadratic programming over ellipsoids with applications to constrained linear regression and tensor decomposition , 2019, Neural Computing and Applications.

[23]  W. Gander,et al.  A Constrained Eigenvalue Problem , 1989 .

[24]  Andrzej Cichocki,et al.  Fast Alternating LS Algorithms for High Order CANDECOMP/PARAFAC Tensor Factorizations , 2013, IEEE Transactions on Signal Processing.

[25]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[26]  Andrzej Cichocki,et al.  Tensor Networks for Latent Variable Analysis. Part I: Algorithms for Tensor Train Decomposition , 2016, ArXiv.

[27]  Andrzej Cichocki,et al.  Partitioned Hierarchical alternating least squares algorithm for CP tensor decomposition , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  R. A. Harshman,et al.  Data preprocessing and the extended PARAFAC model , 1984 .

[31]  Lars Kai Hansen,et al.  Parallel Factor Analysis as an exploratory tool for wavelet transformed event-related EEG , 2006, NeuroImage.

[32]  Alexander J. Smola,et al.  Fast and Guaranteed Tensor Decomposition via Sketching , 2015, NIPS.

[33]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[34]  Andrzej Cichocki,et al.  Numerical CP decomposition of some difficult tensors , 2016, J. Comput. Appl. Math..

[35]  Andrzej Cichocki,et al.  Smooth PARAFAC Decomposition for Tensor Completion , 2015, IEEE Transactions on Signal Processing.

[36]  P. Paatero Construction and analysis of degenerate PARAFAC models , 2000 .

[37]  Lieven De Lathauwer,et al.  Swamp reducing technique for tensor decomposition , 2008, 2008 16th European Signal Processing Conference.

[38]  W. Rayens,et al.  Two-factor degeneracies and a stabilization of PARAFAC , 1997 .

[39]  Ivan V. Oseledets,et al.  Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition , 2014, ICLR.

[40]  Pierre Comon,et al.  Nonnegative approximations of nonnegative tensors , 2009, ArXiv.