Krylov-Levenberg-Marquardt Algorithm for Structured Tucker Tensor Decompositions

Structured Tucker tensor decomposition models complete or incomplete multiway data sets (tensors), where the core tensor and the factor matrices can obey different constraints. The model includes block-term decomposition or canonical polyadic decomposition as special cases. We propose a very flexible optimization method for the structured Tucker decomposition problem, based on the second-order Levenberg-Marquardt optimization, using an approximation of the Hessian matrix by the Krylov subspace method. An algorithm with limited sensitivity of the decomposition is included. The proposed algorithm is shown to perform well in comparison to existing tensor decomposition methods.

[1]  Andrzej Cichocki,et al.  Low rank tensor deconvolution , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Nikos D. Sidiropoulos,et al.  Parallel Algorithms for Constrained Tensor Factorization via Alternating Direction Method of Multipliers , 2014, IEEE Transactions on Signal Processing.

[3]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[4]  Andrzej Cichocki,et al.  Weighted Krylov-Levenberg-Marquardt Method for Canonical Polyadic Tensor Decomposition , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Nikos D. Sidiropoulos,et al.  Tensor Decomposition for Signal Processing and Machine Learning , 2016, IEEE Transactions on Signal Processing.

[6]  Kejun Huang,et al.  Low-complexity Proximal Gauss-Newton Algorithm for Nonnegative Matrix Factorization , 2019, 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[7]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[8]  Lieven De Lathauwer,et al.  Structured Data Fusion , 2015, IEEE Journal of Selected Topics in Signal Processing.

[9]  Sergios Theodoridis,et al.  Blind fMRI source unmixing via higher-order tensor decompositions , 2019, Journal of Neuroscience Methods.

[10]  Andrzej Cichocki,et al.  Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network , 2020, ECCV.

[11]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[12]  J. Landsberg Tensors: Geometry and Applications , 2011 .

[13]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[14]  B. Khoromskij O(dlog N)-Quantics Approximation of N-d Tensors in High-Dimensional Numerical Modeling , 2011 .

[15]  Andrzej Cichocki,et al.  Numerical CP decomposition of some difficult tensors , 2016, J. Comput. Appl. Math..

[16]  VandewalleJoos,et al.  On the Best Rank-1 and Rank-(R1,R2,. . .,RN) Approximation of Higher-Order Tensors , 2000 .

[17]  A. D. de Almeida,et al.  High-order tensor estimation via trains of coupled third-order CP and Tucker decompositions , 2020, Linear Algebra and its Applications.

[18]  R. Harshman,et al.  Modeling multi‐way data with linearly dependent loadings , 2009 .

[19]  D. D. Morrison Methods for nonlinear least squares problems and convergence proofs , 1960 .

[20]  De LathauwerLieven Decompositions of a Higher-Order Tensor in Block TermsPart II: Definitions and Uniqueness , 2008 .

[21]  J. Kruskal,et al.  Candelinc: A general approach to multidimensional analysis of many-way arrays with linear constraints on parameters , 1980 .

[22]  Stefan Handschuh,et al.  Numerical methods in Tensor Networks , 2014 .

[23]  Nico Vervliet,et al.  Numerical Optimization-Based Algorithms for Data Fusion , 2019, Data Handling in Science and Technology.

[24]  Yiu-Ming Cheung,et al.  Rank-One Matrix Completion With Automatic Rank Estimation via L1-Norm Regularization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Athanasios P. Liavas,et al.  Nesterov-Based Alternating Optimization for Nonnegative Tensor Factorization: Algorithm and Parallel Implementation , 2018, IEEE Transactions on Signal Processing.

[26]  Andrzej Cichocki,et al.  A further improvement of a fast damped Gauss-Newton algorithm for candecomp-parafac tensor decomposition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  Qiquan Shi,et al.  Feature Extraction for Incomplete Data Via Low-Rank Tensor Decomposition With Feature Regularization , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[29]  Andrzej Cichocki,et al.  Error Preserving Correction: A Method for CP Decomposition at a Target Error Bound , 2019, IEEE Transactions on Signal Processing.

[30]  Andrzej Cichocki,et al.  Sensitivity in Tensor Decomposition , 2019, IEEE Signal Processing Letters.

[31]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..