Tensor Networks for Big Data Analytics and Large-Scale Optimization Problems

In this paper we review basic and emerging models and associated algorithms for large-scale tensor networks, especially Tensor Train (TT) decompositions using novel mathematical and graphical representations. We discus the concept of tensorization (i.e., creating very high-order tensors from lower-order original data) and super compression of data achieved via quantized tensor train (QTT) networks. The purpose of a tensorization and quantization is to achieve, via low-rank tensor approximations "super" compression, and meaningful, compact representation of structured data. The main objective of this paper is to show how tensor networks can be used to solve a wide class of big data optimization problems (that are far from tractable by classical numerical methods) by applying tensorization and performing all operations using relatively small size matrices and tensors and applying iteratively optimized and approximative tensor contractions. Keywords: Tensor networks, tensor train (TT) decompositions, matrix product states (MPS), matrix product operators (MPO), basic tensor operations, tensorization, distributed representation od data optimization problems for very large-scale problems: generalized eigenvalue decomposition (GEVD), PCA/SVD, canonical correlation analysis (CCA).

[1]  D. Savostyanov,et al.  Exact NMR simulation of protein-size spin systems using tensor train formalism , 2014, 1402.4516.

[2]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[3]  U. Schollwoeck.  The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[4]  Andrew Critch.  Algebraic Geometry of Hidden Markov and Related Models , 2013 .

[5]  Ivan V. Oseledets,et al.  Time Integration of Tensor Trains , 2014, SIAM J. Numer. Anal..

[6]  Daniel Kressner,et al.  Algorithm 941 , 2014 .

[7]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[8]  Renato Pajarola,et al.  TAMRESH – Tensor Approximation Multiresolution Hierarchy for Interactive Volume Visualization , 2013, Comput. Graph. Forum.

[9]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[10]  W. Dur,et al.  Concatenated tensor network states , 2009, 0904.1925.

[11]  Youssef M. Marzouk,et al.  Spectral Tensor-Train Decomposition , 2014, SIAM J. Sci. Comput..

[12]  Z. Y. Xie,et al.  Renormalization of tensor-network states , 2010, 1002.1405.

[13]  Andrzej Cichocki,et al.  Computing Sparse Representations of Multidimensional Signals Using Kronecker Bases , 2013, Neural Computation.

[14]  Ivan V. Oseledets,et al.  Approximation of 2d˟2d Matrices Using Tensor Decomposition , 2010, SIAM J. Matrix Anal. Appl..

[15]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[16]  Jianhua Z. Huang,et al.  Sparse principal component analysis via regularized low rank matrix approximation , 2008 .

[17]  B. Khoromskij.  O(dlog N)-Quantics Approximation of N-d Tensors in High-Dimensional Numerical Modeling , 2011 .

[18]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[19]  Frank Verstraete,et al.  Faster identification of optimal contraction sequences for tensor networks. , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[20]  Tonio Ball,et al.  Multilinear Subspace Regression: An Orthogonal Tensor Decomposition Approach , 2011, NIPS.

[21]  Youssef M. Marzouk,et al.  Spectral Tensor-Train Decomposition for low-rank surrogate models , 2014 .

[22]  F. Verstraete,et al.  Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems , 2008, 0907.2796.

[23]  Vladimir A. Kazeev,et al.  Multilevel Toeplitz Matrices Generated by Tensor-Structured Vectors and Convolution with Logarithmic Complexity , 2013, SIAM J. Sci. Comput..

[24]  Morten Mørup,et al.  Applications of tensor (multiway array) factorizations and decompositions in data mining , 2011, WIREs Data Mining Knowl. Discov..

[25]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[26]  B. Khoromskij.  Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[27]  Christoph Schwab,et al.  Low-rank tensor structure of linear diffusion operators in the TT and QTT formats☆ , 2013 .

[28]  Wolfgang Hackbusch,et al.  An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples , 2011, Comput. Methods Appl. Math..

[29]  S. V. Dolgov,et al.  ALTERNATING MINIMAL ENERGY METHODS FOR LINEAR SYSTEMS IN HIGHER DIMENSIONS∗ , 2014 .

[30]  Jianhua Z. Huang,et al.  The Analysis of Two-Way Functional Data Using Two-Way Regularized Singular Value Decompositions , 2009 .

[31]  Naotaka Fujii,et al.  Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Reinhold Schneider,et al.  Dynamical Approximation by Hierarchical Tucker and Tensor-Train Tensors , 2013, SIAM J. Matrix Anal. Appl..

[33]  F. Verstraete,et al.  Variational numerical renormalization group: bridging the gap between NRG and density matrix renormalization group. , 2011, Physical review letters.

[34]  Haiping Lu,et al.  A survey of multilinear subspace learning for tensor data , 2011, Pattern Recognit..

[35]  Roman Orus,et al.  Exploring corner transfer matrices and corner tensors for the classical simulation of quantum lattice systems , 2011, 1112.4101.

[36]  R. Pfeifer,et al.  NCON: A tensor network contractor for MATLAB , 2014, 1402.0939.

[37]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[38]  Yurii Nesterov,et al.  Generalized Power Method for Sparse Principal Component Analysis , 2008, J. Mach. Learn. Res..

[39]  Eugene E. Tyrtyshnikov,et al.  Tensor-Train Ranks for Matrices and Their Inverses , 2011, Comput. Methods Appl. Math..

[40]  S. Dolgov.  TT-GMRES: solution to a linear system in the structured tensor format , 2012, 1206.5512.

[41]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[42]  P. Kroonenberg.  Applied Multiway Data Analysis , 2008 .

[43]  G. Vidal.  Efficient classical simulation of slightly entangled quantum computations. , 2003, Physical review letters.

[44]  G. Evenbly,et al.  Algorithms for entanglement renormalization , 2007, 0707.1454.

[45]  Jennifer Seberry,et al.  The Strong Kronecker Product , 1994, J. Comb. Theory, Ser. A.

[46]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[47]  Boris N. Khoromskij,et al.  Computation of extreme eigenvalues in higher dimensions using block tensor train format , 2013, Comput. Phys. Commun..

[48]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[49]  E. M. Stoudenmire,et al.  ITensor Library Release v0.2.5 , 2014 .

[50]  Eugene E. Tyrtyshnikov,et al.  Algebraic Wavelet Transform via Quantics Tensor Train Decomposition , 2011, SIAM J. Sci. Comput..

[51]  T. Schulte-Herbrüggen,et al.  Computations in quantum tensor networks , 2012, 1212.5005.

[52]  Andrzej Cichocki,et al.  Fundamental Tensor Operations for Large-Scale Data Analysis in Tensor Train Formats , 2014, ArXiv.

[53]  A. Uschmajew,et al.  On low-rank approximability of solutions to high-dimensional operator equations and eigenvalue problems , 2014, 1406.7026.

[54]  Vladimir A. Kazeev,et al.  Direct Solution of the Chemical Master Equation Using Quantized Tensor Trains , 2014, PLoS Comput. Biol..

[55]  N. Alexander,et al.  Putting MRFs on a Tensor Train , 2014 .

[56]  Genevera I. Allen.  Regularized Tensor Factorizations and Higher-Order Principal Components Analysis , 2012, 1202.2476.

[57]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[58]  Fuzhen Zhang,et al.  A matrix decomposition and its applications , 2015 .

[59]  Naotaka Fujii,et al.  Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Andrzej Cichocki,et al.  Extended HALS algorithm for nonnegative Tucker decomposition and its applications for multiway analysis and classification , 2011, Neurocomputing.

[61]  Chao Yang,et al.  Trace-Penalty Minimization for Large-Scale Eigenspace Computation , 2016, J. Sci. Comput..

[62]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[63]  VLADIMIR A. KAZEEV,et al.  Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse , 2012, SIAM J. Matrix Anal. Appl..

[64]  Andrzej Cichocki,et al.  Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions , 2014, ArXiv.

[65]  S Montangero,et al.  Quantum multiscale entanglement renormalization ansatz channels. , 2008, Physical review letters.

[66]  N.D. Sidiropoulos,et al.  Low-rank decomposition of multi-way arrays: a signal processing perspective , 2004, Processing Workshop Proceedings, 2004 Sensor Array and Multichannel Signal.

[67]  Andrzej Cichocki,et al.  Efficient Nonnegative Tucker Decompositions: Algorithms and Uniqueness , 2014, IEEE Transactions on Image Processing.

[68]  Subir Sachdev.  Viewpoint: Tensor networks—a new tool for old problems , 2009 .

[69]  Daniel Kressner,et al.  Low-Rank Tensor Methods with Subspace Correction for Symmetric Eigenvalue Problems , 2014, SIAM J. Sci. Comput..

[70]  U. Schollwöck.  Matrix Product State Algorithms: DMRG, TEBD and Relatives , 2013 .

[71]  Boris N. Khoromskij,et al.  Two-Level QTT-Tucker Format for Optimized Tensor Calculus , 2013, SIAM J. Matrix Anal. Appl..

[72]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[73]  Andrzej Cichocki,et al.  PARAFAC algorithms for large-scale problems , 2011, Neurocomputing.

[74]  Yu-Kun Huang.  Biorthonormal transfer-matrix renormalization-group method for non-Hermitian matrices. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[75]  A. Cichocki,et al.  Generalizing the column–row matrix decomposition to multi-way arrays , 2010 .

[76]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[77]  Jianhua Z. Huang,et al.  Biclustering via Sparse Singular Value Decomposition , 2010, Biometrics.

[78]  A. Cichocki.  脳機能計測と生体信号入出力(第7回)Tensor Decompositions: New Concepts in Brain Data Analysis? , 2011 .

[79]  Stefan Handschuh.  Changing the topology of Tensor Networks , 2012 .

[80]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[81]  Frank Verstraete,et al.  Matrix product state representations , 2006, Quantum Inf. Comput..

[82]  Bart Vandereycken,et al.  The geometry of algorithms using hierarchical tensors , 2013, Linear Algebra and its Applications.

[83]  Andrew Critch,et al.  Algebraic Geometry of Matrix Product States , 2012, 1210.2812.

[84]  Rasmus Bro,et al.  Multi-way Analysis with Applications in the Chemical Sciences , 2004 .

[85]  Yousef Saad,et al.  Trace optimization and eigenproblems in dimension reduction methods , 2011, Numer. Linear Algebra Appl..

[86]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[87]  J. Ignacio Cirac,et al.  Unifying projected entangled pair state contractions , 2013, 1311.6696.

[88]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[89]  Ivan V. Oseledets,et al.  DMRG Approach to Fast Linear Algebra in the TT-Format , 2011, Comput. Methods Appl. Math..

[90]  N. Ahuja,et al.  Out-of-core tensor approximation of multi-dimensional matrices of visual data , 2005, SIGGRAPH 2005.

[91]  Genevera I. Allen,et al.  Sparse non-negative generalized PCA with applications to metabolomics , 2011, Bioinform..

[92]  Andrzej Cichocki,et al.  On Revealing Replicating Structures in Multiway Data: A Novel Tensor Decomposition Approach , 2012, LVA/ICA.

[93]  A. Cichocki,et al.  Tensor decompositions for feature extraction and classification of high dimensional datasets , 2010 .

[94]  R. Tibshirani,et al.  A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. , 2009, Biostatistics.

[95]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .