Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1

Machine learning and data mining algorithms are becoming increasingly important in analyzing large volume, multi-relational and ∗Copyright A.Cichocki et al. Please make reference to: A. Cichocki, N. Lee, I. Oseledets, A.-H. Phan, Q. Zhao and D.P. Mandic (2016), “Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions”, Foundations and Trends in Machine Learning: Vol. 9: No. 4-5, pp 249-429.

[1]  A. Cichocki,et al.  Tensor decompositions for feature extraction and classification of high dimensional datasets , 2010 .

[2]  Raf Vandebril,et al.  A New Truncation Strategy for the Higher-Order Singular Value Decomposition , 2012, SIAM J. Sci. Comput..

[3]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[4]  Ivan V. Oseledets,et al.  Approximation of 2d˟2d Matrices Using Tensor Decomposition , 2010, SIAM J. Matrix Anal. Appl..

[5]  Tao Wu,et al.  General Tensor Spectral Co-clustering for Higher-Order Data , 2016, NIPS.

[6]  Boris N. Khoromskij,et al.  Computation of extreme eigenvalues in higher dimensions using block tensor train format , 2013, Comput. Phys. Commun..

[7]  Peter Richtárik,et al.  Parallel coordinate descent methods for big data optimization , 2012, Mathematical Programming.

[8]  Zenglin Xu,et al.  Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis , 2011, ICML.

[9]  R. Schneider,et al.  Tree Tensor Network State with Variable Tensor Order: An Efficient Multireference Method for Strongly Correlated Systems , 2015, Journal of chemical theory and computation.

[10]  Daniel Kressner,et al.  Algorithm 941 , 2014 .

[11]  Lars Karlsson,et al.  Parallel algorithms for tensor completion in the CP format , 2016, Parallel Comput..

[12]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[13]  Mario Bebendorf,et al.  Wideband nested cross approximation for Helmholtz problems , 2015, Numerische Mathematik.

[14]  Lieven De Lathauwer,et al.  Blind Separation of Exponential Polynomials and the Decomposition of a Tensor in Rank-(Lr, Lr, 1) Terms , 2011, SIAM J. Matrix Anal. Appl..

[15]  André Uschmajew,et al.  Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation , 2012, SIAM J. Matrix Anal. Appl..

[16]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[17]  Garnet Kin-Lic Chan,et al.  Efficient tree tensor network states (TTNS) for quantum chemistry: generalizations of the density matrix renormalization group algorithm. , 2013, The Journal of chemical physics.

[18]  Nico Vervliet,et al.  A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors , 2016, IEEE Journal of Selected Topics in Signal Processing.

[19]  Pierre Comon,et al.  Multiarray Signal Processing: Tensor decomposition meets compressed sensing , 2010, ArXiv.

[20]  Zenglin Xu,et al.  Distributed Flexible Nonlinear Tensor Factorization , 2016, NIPS.

[21]  Wei Chu,et al.  Probabilistic Models for Incomplete Multi-dimensional Arrays , 2009, AISTATS.

[22]  Andrzej Cichocki,et al.  Robust Multilinear Tensor Rank Estimation Using Higher Order Singular Value Decomposition and Information Criteria , 2017, IEEE Transactions on Signal Processing.

[23]  Martin J. Wainwright,et al.  Asymptotic behavior of ℓp-based Laplacian regularization in semi-supervised learning , 2016, ArXiv.

[24]  Pierre-Antoine Absil,et al.  RTRMC: A Riemannian trust-region method for low-rank matrix completion , 2011, NIPS.

[25]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[26]  Andrzej Cichocki,et al.  Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions , 2014, ArXiv.

[27]  Alexander J. Smola,et al.  Fast and Guaranteed Tensor Decomposition via Sketching , 2015, NIPS.

[28]  David F. Gleich,et al.  Scalable Methods for Nonnegative Matrix Factorizations of Near-separable Tall-and-skinny Matrices , 2014, NIPS.

[29]  R. Pfeifer,et al.  NCON: A tensor network contractor for MATLAB , 2014, 1402.0939.

[30]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[31]  Dario Andrea Bini.  Tensor and border rank of certain classes of matrices and the fast evaluation of determinant inverse matrix and eigenvalues , 1985 .

[32]  Tae-Kyun Kim,et al.  Tensor Canonical Correlation Analysis for Action Classification , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Michael Griebel,et al.  Data Mining with Sparse Grids , 2001, Computing.

[34]  I. McCulloch,et al.  Strictly single-site DMRG algorithm with subspace expansion , 2015, 1501.05504.

[35]  S. Dolgov.  TT-GMRES: solution to a linear system in the structured tensor format , 2012, 1206.5512.

[36]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[37]  P. Kroonenberg.  Applied Multiway Data Analysis , 2008 .

[38]  Percy Liang,et al.  Tensor Factorization via Matrix Factorization , 2015, AISTATS.

[39]  Andrzej Cichocki,et al.  Canonical Polyadic Decomposition Based on a Single Mode Blind Source Separation , 2012, IEEE Signal Processing Letters.

[40]  Pierre-Antoine Absil,et al.  Low-rank retractions: a survey and new results , 2015, Comput. Optim. Appl..

[41]  Lloyd N. Trefethen,et al.  Cubature, Approximation, and Isotropy in the Hypercube , 2017, SIAM Rev..

[42]  J. Ross Beveridge,et al.  Action classification on product manifolds , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[44]  Gérard Favier,et al.  Overview of tensor decompositions with applications to communications , 2016 .

[45]  Boris N. Khoromskij,et al.  Simultaneous state-time approximation of the chemical master equation using tensor product formats , 2015, Numer. Linear Algebra Appl..

[46]  E. M. Stoudenmire,et al.  ITensor Library Release v0.2.5 , 2014 .

[47]  David F. Gleich,et al.  Multilinear PageRank , 2014, SIAM J. Matrix Anal. Appl..

[48]  Eugene E. Tyrtyshnikov,et al.  Algebraic Wavelet Transform via Quantics Tensor Train Decomposition , 2011, SIAM J. Sci. Comput..

[49]  Michael I. Jordan,et al.  A Probabilistic Interpretation of Canonical Correlation Analysis , 2005 .

[50]  T. Schulte-Herbrüggen,et al.  Computations in quantum tensor networks , 2012, 1212.5005.

[51]  Volker Tresp,et al.  Learning with Memory Embeddings , 2015, ArXiv.

[52]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[53]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[54]  Stefan Klus,et al.  Towards tensor-based methods for the numerical approximation of the Perron-Frobenius and Koopman operator , 2015, 1512.06527.

[55]  Nico Vervliet,et al.  Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis , 2014, IEEE Signal Processing Magazine.

[56]  B. Khoromskij.  Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[57]  Christoph Schwab,et al.  Low-rank tensor structure of linear diffusion operators in the TT and QTT formats☆ , 2013 .

[58]  Yongxin Yang,et al.  Deep Multi-task Representation Learning: A Tensor Factorisation Approach , 2016, ICLR.

[59]  Andrzej Cichocki,et al.  Tensor Networks for Big Data Analytics and Large-Scale Optimization Problems , 2014, ArXiv.

[60]  R. Tibshirani,et al.  A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. , 2009, Biostatistics.

[61]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[62]  G. Vidal,et al.  Classical simulation of quantum many-body systems with a tree tensor network , 2005, quant-ph/0511070.

[63]  Michael K. Ng,et al.  Sparse Canonical Correlation Analysis: New Formulation and Algorithm , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Klaus-Robert Müller,et al.  Three-Way Analysis of Spectrospatial Electromyography Data: Classification and Interpretation , 2015, PloS one.

[65]  Michael W. Mahoney,et al.  Spectral Gap Error Bounds for Improving CUR Matrix Decomposition and the Nyström Method , 2015, AISTATS.

[66]  Lieven De Lathauwer,et al.  Blind Signal Separation via Tensor Decomposition With Vandermonde Factor: Canonical Polyadic Decomposition , 2013, IEEE Transactions on Signal Processing.

[67]  Lieven De Lathauwer,et al.  Exact line and plane search for tensor optimization , 2013, Computational optimization and applications.

[68]  M. Reiher,et al.  Quantum-information analysis of electronic states of different molecular structures , 2010, 1008.4607.

[69]  Andrzej Cichocki,et al.  Fast Nonnegative Matrix/Tensor Factorization Based on Low-Rank Approximation , 2012, IEEE Transactions on Signal Processing.

[70]  David J. Schwab,et al.  Supervised Learning with Quantum-Inspired Tensor Networks , 2016, ArXiv.

[71]  M. Steinlechner.  Riemannian Optimization for Solving High-Dimensional Problems with Low-Rank Tensor Structure , 2016 .

[72]  Vishal Monga,et al.  Robust Video Hashing via Multilinear Subspace Projections , 2012, IEEE Transactions on Image Processing.

[73]  Daniel M. Dunlavy,et al.  A scalable optimization approach for fitting canonical tensor decompositions , 2011 .

[74]  Andrzej Cichocki,et al.  Tensor Networks for Latent Variable Analysis. Part I: Algorithms for Tensor Train Decomposition , 2016, ArXiv.

[75]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[76]  Stefan Handschuh.  Numerical methods in Tensor Networks , 2014 .

[77]  Andrzej Cichocki,et al.  Low rank tensor deconvolution , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[78]  Ivan V. Oseledets,et al.  Fast Multidimensional Convolution in Low-Rank Tensor Formats via Cross Approximation , 2015, SIAM J. Sci. Comput..

[79]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[80]  Andrzej Cichocki,et al.  PARAFAC algorithms for large-scale problems , 2011, Neurocomputing.

[81]  Nikos D. Sidiropoulos,et al.  Generalizing Carathéodory's uniqueness of harmonic parameterization to N dimensions , 2001, IEEE Trans. Inf. Theory.

[82]  Edoardo Di Napoli,et al.  Towards an efficient use of the BLAS library for multilinear tensor contractions , 2013, Appl. Math. Comput..

[83]  Felix J. Herrmann,et al.  Optimization on the Hierarchical Tucker manifold – Applications to tensor completion , 2014, Linear Algebra and its Applications.

[84]  Daniel Kressner,et al.  Multigrid methods combined with low-rank approximation for tensor structured Markov chains , 2016 .

[85]  Lieven De Lathauwer,et al.  Löwner-Based Blind Signal Separation of Rational Functions With Applications , 2016, IEEE Transactions on Signal Processing.

[86]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[87]  Pierre Vandergheynst,et al.  Tensor low-rank and sparse light field photography , 2016, Comput. Vis. Image Underst..

[88]  Jeff M. Phillips,et al.  Improved Practical Matrix Sketching with Guarantees , 2014, IEEE Transactions on Knowledge and Data Engineering.

[89]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[90]  Zhihua Zhang,et al.  Improving CUR matrix decomposition and the Nyström approximation via adaptive sampling , 2013, J. Mach. Learn. Res..

[91]  Boris N. Khoromskij,et al.  Superfast Wavelet Transform Using Quantics-TT Approximation. I. Application to Haar Wavelets , 2014, Comput. Methods Appl. Math..

[92]  White,et al.  Density-matrix algorithms for quantum renormalization groups. , 1993, Physical review. B, Condensed matter.

[93]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[94]  Nikos D. Sidiropoulos,et al.  Parallel Algorithms for Constrained Tensor Factorization via Alternating Direction Method of Multipliers , 2014, IEEE Transactions on Signal Processing.

[95]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[96]  G. Evenbly,et al.  Algorithms for entanglement renormalization , 2007, 0707.1454.

[97]  Christine Tobler,et al.  Low-rank tensor methods for linear systems and eigenvalue problems , 2012 .

[98]  Xiu Yang,et al.  Enabling High-Dimensional Hierarchical Uncertainty Quantification by ANOVA and Tensor-Train Decomposition , 2014, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[99]  Ivan V. Oseledets,et al.  DMRG Approach to Fast Linear Algebra in the TT-Format , 2011, Comput. Methods Appl. Math..

[100]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[101]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[102]  Jimeng Sun,et al.  An input-adaptive and in-place approach to dense tensor-times-matrix multiply , 2015, SC15: International Conference for High Performance Computing, Networking, Storage and Analysis.

[103]  Arie Yeredor,et al.  Joint Matrices Decompositions and Blind Source Separation: A survey of methods, identification, and applications , 2014, IEEE Signal Processing Magazine.

[104]  Xuelong Li,et al.  General Tensor Discriminant Analysis and Gabor Features for Gait Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[105]  Aapo Hyvärinen,et al.  Independent component analysis of fMRI group studies by self-organizing clustering , 2005, NeuroImage.

[106]  Nikos D. Sidiropoulos,et al.  Tensors for Data Mining and Data Fusion , 2016, ACM Trans. Intell. Syst. Technol..

[107]  Oded Schwartz,et al.  Improving the Numerical Stability of Fast Matrix Multiplication , 2015, SIAM J. Matrix Anal. Appl..

[108]  Weichung Wang,et al.  Integrating multiple random sketches for singular value decomposition , 2016, 1608.08285.

[109]  Andrzej Cichocki,et al.  Very Large-Scale Singular Value Decomposition Using Tensor Train Networks , 2014, ArXiv.

[110]  Vladimir A. Kazeev,et al.  Multilevel Toeplitz Matrices Generated by Tensor-Structured Vectors and Convolution with Logarithmic Complexity , 2013, SIAM J. Sci. Comput..

[111]  Alejandro C Olivieri,et al.  Analytical advantages of multivariate data processing. One, two, three, infinity? , 2008, Analytical chemistry.

[112]  Junbin Gao,et al.  Heterogeneous Tensor Decomposition for Clustering via Manifold Optimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[113]  Boris N. Khoromskij,et al.  Fast tensor method for summation of long-range potentials on 3D lattices with defects , 2016, Numer. Linear Algebra Appl..

[114]  Reinhold Schneider,et al.  Dynamical Approximation by Hierarchical Tucker and Tensor-Train Tensors , 2013, SIAM J. Matrix Anal. Appl..

[115]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[116]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[117]  Matthias Bolten,et al.  Multigrid Methods for Tensor Structured Markov Chains with Low Rank Approximation , 2014, SIAM J. Sci. Comput..

[118]  VLADIMIR A. KAZEEV,et al.  Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse , 2012, SIAM J. Matrix Anal. Appl..

[119]  Pierre Comon,et al.  Tensor CP Decomposition With Structured Factor Matrices: Algorithms and Performance , 2016, IEEE Journal of Selected Topics in Signal Processing.

[120]  Petros Drineas,et al.  Tensor-CUR decompositions for tensor-based data , 2006, KDD '06.

[121]  Michael W. Mahoney,et al.  A randomized algorithm for a tensor-based generalization of the singular value decomposition , 2007 .

[122]  Bülent Yener,et al.  Unsupervised Multiway Data Analysis: A Literature Survey , 2009, IEEE Transactions on Knowledge and Data Engineering.

[123]  Anton Rodomanov,et al.  Putting MRFs on a Tensor Train , 2014, ICML.

[124]  Andrzej Cichocki,et al.  Big Data Matrix Singular Value Decomposition Based on Low-Rank Tensor Train Decomposition , 2014, ISNN.

[125]  G. Ehlers,et al.  Entanglement structure of the Hubbard model in momentum space , 2015, 1508.07477.

[126]  Hadi Fanaee-T,et al.  Tensor-based anomaly detection: An interdisciplinary survey , 2016, Knowl. Based Syst..

[127]  Yong Luo,et al.  Tensor Canonical Correlation Analysis for Multi-View Dimension Reduction , 2015, IEEE Transactions on Knowledge and Data Engineering.

[128]  Bamdev Mishra,et al.  Low-Rank Optimization with Trace Norm Penalty , 2011, SIAM J. Optim..

[129]  Christos Faloutsos,et al.  GigaTensor: scaling tensor analysis up by 100 times - algorithms and discoveries , 2012, KDD.

[130]  J. Ogutu,et al.  Regularized group regression methods for genomic prediction: Bridge, MCP, SCAD, group bridge, group lasso, sparse group lasso, group MCP and group SCAD , 2014, BMC Proceedings.

[131]  D. Pérez-García,et al.  PEPS as ground states: Degeneracy and topology , 2010, 1001.3807.

[132]  Ngai Wong,et al.  A constructive arbitrary‐degree Kronecker product decomposition of tensors , 2015, Numer. Linear Algebra Appl..

[133]  Nikos D. Sidiropoulos,et al.  Tensor Decomposition for Signal Processing and Machine Learning , 2016, IEEE Transactions on Signal Processing.

[134]  Morten Mørup,et al.  Applications of tensor (multiway array) factorizations and decompositions in data mining , 2011, WIREs Data Mining Knowl. Discov..

[135]  Jianhua Z. Huang,et al.  The Analysis of Two-Way Functional Data Using Two-Way Regularized Singular Value Decompositions , 2009 .

[136]  Oded Schwartz,et al.  Hypergraph Partitioning for Parallel Sparse Matrix-Matrix Multiplication , 2015, SPAA.

[137]  Dong Wang,et al.  Efficient dimension reduction for high-dimensional matrix-valued data , 2016, Neurocomputing.

[138]  F. Verstraete,et al.  Variational numerical renormalization group: bridging the gap between NRG and density matrix renormalization group. , 2011, Physical review letters.

[139]  Christos Faloutsos,et al.  HaTen2: Billion-scale tensor decompositions , 2015, 2015 IEEE 31st International Conference on Data Engineering.

[140]  Aapo Hyvärinen,et al.  Group-PCA for very large fMRI datasets , 2014, NeuroImage.

[141]  Chris H. Q. Ding,et al.  Simultaneous tensor subspace selection and clustering: the equivalence of high order svd and k-means clustering , 2008, KDD.

[142]  A. Uschmajew,et al.  On low-rank approximability of solutions to high-dimensional operator equations and eigenvalue problems , 2014, 1406.7026.

[143]  Vladimir A. Kazeev,et al.  Direct Solution of the Chemical Master Equation Using Quantized Tensor Trains , 2014, PLoS Comput. Biol..

[144]  Jianhua Z. Huang,et al.  Biclustering via Sparse Singular Value Decomposition , 2010, Biometrics.

[145]  André Uschmajew,et al.  On the interconnection between the higher-order singular values of real tensors , 2017, Numerische Mathematik.

[146]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[147]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[148]  P. Tichavsky,et al.  Fast Approximate Joint Diagonalization Incorporating Weight Matrices , 2009, IEEE Transactions on Signal Processing.

[149]  Amnon Shashua,et al.  Convolutional Rectifier Networks as Generalized Tensor Decompositions , 2016, ICML.

[150]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[151]  Liqing Zhang,et al.  Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[152]  Stefan Ragnarsson-Torbergsen.  Structured Tensor Computations: Blocking, Symmetries And Kronecker Factorizations , 2012 .

[153]  Rasmus Pagh,et al.  Fast and scalable polynomial kernels via explicit feature maps , 2013, KDD.

[154]  David F. Gleich,et al.  Model Reduction With MapReduce-enabled Tall and Skinny Singular Value Decomposition , 2013, SIAM J. Sci. Comput..

[155]  Andrzej Cichocki,et al.  Stable, Robust, and Super Fast Reconstruction of Tensors Using Multi-Way Projections , 2014, IEEE Transactions on Signal Processing.

[156]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[157]  Michael W. Mahoney,et al.  Future Directions in Tensor-Based Computation and Modeling , 2009 .

[158]  Mario Bebendorf,et al.  Adaptive Cross Approximation of Multivariate Functions , 2011 .

[159]  Martin Stoll,et al.  Fast tensor product solvers for optimization problems with fractional differential equations as constraints , 2016, Appl. Math. Comput..

[160]  Jimeng Sun,et al.  Beyond streams and graphs: dynamic tensor analysis , 2006, KDD '06.

[161]  Babak Hossein Khalaj,et al.  A unified approach to sparse signal processing , 2009, EURASIP Journal on Advances in Signal Processing.

[162]  Liqing Zhang,et al.  Kernelization of Tensor-Based Models for Multiway Data Analysis: Processing of Multidimensional Structured Data , 2013, IEEE Signal Processing Magazine.

[163]  Stefan Handschuh.  Changing the topology of Tensor Networks , 2012 .

[164]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[165]  Petros Drineas,et al.  CUR matrix decompositions for improved data analysis , 2009, Proceedings of the National Academy of Sciences.

[166]  Michel Verhaegen,et al.  Learning multidimensional Fourier series with tensor trains , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[167]  Andrzej Cichocki,et al.  Computing Sparse Representations of Multidimensional Signals Using Kronecker Bases , 2013, Neural Computation.

[168]  Visa Koivunen,et al.  Sequential Unfolding SVD for Tensors With Applications in Array Signal Processing , 2009, IEEE Transactions on Signal Processing.

[169]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[170]  Michael Ulbrich,et al.  Constrained Optimization with Low-Rank Tensors and Applications to Parametric Problems with PDEs , 2017, SIAM J. Sci. Comput..

[171]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[172]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[173]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[174]  Mark W. Woolrich,et al.  Linked independent component analysis for multimodal data fusion , 2011, NeuroImage.

[175]  Nikos D. Sidiropoulos,et al.  Robust iterative fitting of multilinear models , 2005, IEEE Transactions on Signal Processing.

[176]  F. Verstraete,et al.  Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems , 2008, 0907.2796.

[177]  Frank Verstraete,et al.  Matrix product state representations , 2006, Quantum Inf. Comput..

[178]  Bart Vandereycken,et al.  The geometry of algorithms using hierarchical tensors , 2013, Linear Algebra and its Applications.

[179]  Andrew Critch,et al.  Algebraic Geometry of Matrix Product States , 2012, 1210.2812.

[180]  Yousef Saad,et al.  Trace optimization and eigenproblems in dimension reduction methods , 2011, Numer. Linear Algebra Appl..

[181]  Andrzej Cichocki,et al.  Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions , 2015, SIAM J. Matrix Anal. Appl..

[182]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[183]  Junzhou Huang,et al.  Preconditioning for Accelerated Iteratively Reweighted Least Squares in Structured Sparsity Reconstruction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[184]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[185]  Andrzej Cichocki,et al.  Group Component Analysis for Multiblock Data: Common and Individual Feature Extraction , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[186]  S. V. Dolgov,et al.  ALTERNATING MINIMAL ENERGY METHODS FOR LINEAR SYSTEMS IN HIGHER DIMENSIONS∗ , 2014 .

[187]  David Dunson,et al.  Bayesian Factorizations of Big Sparse Tensors , 2013, Journal of the American Statistical Association.

[188]  Aapo Hyvärinen,et al.  Independent component analysis: recent advances , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[189]  Uwe Helmke,et al.  Riemannian Optimization on Tensor Products of Grassmann Manifolds: Applications to Generalized Rayleigh-Quotients , 2010, SIAM J. Matrix Anal. Appl..

[190]  Haiping Lu,et al.  A survey of multilinear subspace learning for tensor data , 2011, Pattern Recognit..

[191]  Andrzej Cichocki,et al.  Fundamental Tensor Operations for Large-Scale Data Analysis in Tensor Train Formats , 2014, ArXiv.

[192]  Volkan Cevher,et al.  Technical Report No . 201 701 January 201 7 RANDOMIZED SINGLE-VIEW ALGORITHMS FOR LOW-RANK MATRIX APPROXIMATION , 2016 .

[193]  Haobin Wang,et al.  Multilayer formulation of the multiconfiguration time-dependent Hartree theory , 2003 .

[194]  Z. Y. Xie,et al.  Renormalization of tensor-network states , 2010, 1002.1405.

[195]  Sergio Cruces,et al.  Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences , 2014, Entropy.

[196]  Nikos D. Sidiropoulos,et al.  From K-Means to Higher-Way Co-Clustering: Multilinear Decomposition With Sparse Latent Factors , 2013, IEEE Transactions on Signal Processing.

[197]  Geoffrey E. Hinton,et al.  Tensor Analyzers , 2013, ICML.

[198]  I. Oseledets.  Constructive Representation of Functions in Low-Rank Tensor Formats , 2012, Constructive Approximation.

[199]  Nikos D. Sidiropoulos,et al.  Parallel factor analysis in sensor array processing , 2000, IEEE Trans. Signal Process..

[200]  Andrzej Cichocki,et al.  From basis components to complex structural patterns , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[201]  Wolfgang Hackbusch,et al.  An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples , 2011, Comput. Methods Appl. Math..

[202]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[203]  H. Matsueda.  Analytic Optimization of a MERA network and its Relevance to Quantum Integrability and Wavelet , 2016, 1608.02205.

[204]  Gordon Wetzstein,et al.  Tensor displays , 2012, ACM Trans. Graph..

[205]  Christos Faloutsos,et al.  Mining billion-scale tensors: algorithms and discoveries , 2016, The VLDB Journal.

[206]  Y. Kao,et al.  Uni10: an open-source library for tensor network algorithms , 2015 .

[207]  L. Mirsky.  SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMS , 1960 .

[208]  Xiao-Gang Wen,et al.  Tensor-product representations for string-net condensed states , 2008, 0809.2821.

[209]  Andrzej Cichocki,et al.  Efficient Nonnegative Tucker Decompositions: Algorithms and Uniqueness , 2014, IEEE Transactions on Image Processing.

[210]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[211]  Daniel Kressner,et al.  Low-Rank Tensor Methods for Communicating Markov Processes , 2014, QEST.

[212]  Lars Grasedyck,et al.  Tree Adaptive Approximation in the Hierarchical Tensor Format , 2014, SIAM J. Sci. Comput..

[213]  L. Lathauwer,et al.  Canonical Polyadic Decomposition with Orthogonality Constraints , 2012 .

[214]  Lieven De Lathauwer,et al.  A Tensor-Based Method for Large-Scale Blind Source Separation Using Segmentation , 2017, IEEE Transactions on Signal Processing.

[215]  Michael I. Jordan,et al.  Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces , 2004, J. Mach. Learn. Res..

[216]  Ivan V. Oseledets,et al.  A low-rank approach to the computation of path integrals , 2015, J. Comput. Phys..

[217]  N.D. Sidiropoulos,et al.  Low-rank decomposition of multi-way arrays: a signal processing perspective , 2004, Processing Workshop Proceedings, 2004 Sensor Array and Multichannel Signal.

[218]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[219]  Trac D. Tran,et al.  Tensor sparsification via a bound on the spectral norm of random tensors , 2010, ArXiv.

[220]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[221]  J. Ignacio Cirac,et al.  Unifying projected entangled pair state contractions , 2013, 1311.6696.

[222]  Edo Liberty,et al.  Efficient Frequent Directions Algorithm for Sparse Matrices , 2016, KDD.

[223]  Zoubin Ghahramani,et al.  Unifying linear dimensionality reduction , 2014, 1406.0873.

[224]  Tamara G. Kolda,et al.  MATLAB tensor classes for fast algorithm prototyping. , 2004 .

[225]  Demetri Terzopoulos,et al.  Multilinear Analysis of Image Ensembles: TensorFaces , 2002, ECCV.

[226]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[227]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[228]  Genevera I. Allen,et al.  Sparse non-negative generalized PCA with applications to metabolomics , 2011, Bioinform..

[229]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[230]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[231]  Andrzej Cichocki,et al.  Fast and unique Tucker decompositions via multiway blind source separation , 2012 .

[232]  Andrzej Cichocki,et al.  On Revealing Replicating Structures in Multiway Data: A Novel Tensor Decomposition Approach , 2012, LVA/ICA.

[233]  Andrew Critch.  Algebraic Geometry of Hidden Markov and Related Models , 2013 .

[234]  Ivan V. Oseledets,et al.  Time Integration of Tensor Trains , 2014, SIAM J. Numer. Anal..

[235]  Tonio Ball,et al.  Multilinear Subspace Regression: An Orthogonal Tensor Decomposition Approach , 2011, NIPS.

[236]  Michael Steinlechner,et al.  Riemannian Optimization for High-Dimensional Tensor Completion , 2016, SIAM J. Sci. Comput..

[237]  Daniel Kressner,et al.  Preconditioned Low-rank Riemannian Optimization for Linear Systems with Tensor Product Structure , 2015, SIAM J. Sci. Comput..

[238]  Ngai Wong,et al.  A Constructive Algorithm for Decomposing a Tensor into a Finite Sum of Orthonormal Rank-1 Terms , 2014, SIAM J. Matrix Anal. Appl..

[239]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[240]  A. Yu. Mikhalev,et al.  Iterative representing set selection for nested cross approximation , 2013, Numer. Linear Algebra Appl..

[241]  Gene H. Golub,et al.  Symmetric Tensors and Symmetric Tensor Rank , 2008, SIAM J. Matrix Anal. Appl..

[242]  S. Dolgov.  Tensor product methods in numerical simulation of high-dimensional dynamical problems , 2014 .

[243]  Fan Yang,et al.  Tensor and its tucker core: The invariance relationships , 2016, Numer. Linear Algebra Appl..

[244]  David F. Gleich,et al.  Tall and skinny QR factorizations in MapReduce architectures , 2011, MapReduce '11.

[245]  Tae-Kyun Kim,et al.  Canonical Correlation Analysis of Video Volume Tensors for Action Categorization and Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[246]  S Montangero,et al.  Quantum multiscale entanglement renormalization ansatz channels. , 2008, Physical review letters.

[247]  Pierre-Antoine Absil,et al.  Robust Low-Rank Matrix Completion by Riemannian Optimization , 2016, SIAM J. Sci. Comput..

[248]  Florian Roemer,et al.  Higher-Order SVD-Based Subspace Estimation to Improve the Parameter Estimation Accuracy in Multidimensional Harmonic Retrieval Problems , 2008, IEEE Transactions on Signal Processing.

[249]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[250]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[251]  J. H. Choi,et al.  DFacTo: Distributed Factorization of Tensors , 2014, NIPS.

[252]  A. Cichocki,et al.  Generalizing the column–row matrix decomposition to multi-way arrays , 2010 .

[253]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[254]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[255]  André Lima Férrer de Almeida,et al.  Overview of constrained PARAFAC models , 2014, EURASIP J. Adv. Signal Process..

[256]  Zhi-Quan Luo,et al.  A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing , 2015, IEEE Signal Processing Magazine.

[257]  Reinhold Schneider,et al.  Tensor Networks and Hierarchical Tensors for the Solution of High-Dimensional Partial Differential Equations , 2016, Found. Comput. Math..

[258]  Andrzej Cichocki,et al.  Fundamental tensor operations for large-scale data analysis using tensor network formats , 2018, Multidimens. Syst. Signal Process..

[259]  Sheng Luo,et al.  Population Value Decomposition, a Framework for the Analysis of Image Populations , 2011, Journal of the American Statistical Association.

[260]  Daniel Kressner,et al.  Low-Rank Tensor Methods with Subspace Correction for Symmetric Eigenvalue Problems , 2014, SIAM J. Sci. Comput..

[261]  Gilles Meyer.  Geometric optimization algorithms for linear regression on fixed-rank matrices , 2011 .

[262]  Lars Grasedyck,et al.  A Review on Adaptive Low-Rank Approximation Techniques in the Hierarchical Tensor Format , 2014 .

[263]  U. Schollwöck.  Matrix Product State Algorithms: DMRG, TEBD and Relatives , 2013 .

[264]  Boris N. Khoromskij,et al.  Two-Level QTT-Tucker Format for Optimized Tensor Calculus , 2013, SIAM J. Matrix Anal. Appl..

[265]  Baba C. Vemuri,et al.  On A Nonlinear Generalization of Sparse Coding and Dictionary Learning , 2013, ICML.

[266]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[267]  D. Savostyanov,et al.  Exact NMR simulation of protein-size spin systems using tensor train formalism , 2014, 1402.4516.

[268]  Visa Koivunen,et al.  New robust LASSO method based on ranks , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[269]  Renato Pajarola,et al.  TAMRESH – Tensor Approximation Multiresolution Hierarchy for Interactive Volume Visualization , 2013, Comput. Graph. Forum.

[270]  Andrzej Cichocki,et al.  Smooth PARAFAC Decomposition for Tensor Completion , 2015, IEEE Transactions on Signal Processing.

[271]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[272]  B. Khoromskij,et al.  Bethe-Salpeter excitation energies using low-rank tensor factorizations , 2015, 1505.02696.

[273]  W. Dur,et al.  Concatenated tensor network states , 2009, 0904.1925.

[274]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[275]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[276]  Yurii Nesterov,et al.  Subgradient methods for huge-scale optimization problems , 2014, Math. Program..

[277]  Boris N. Khoromskij,et al.  Efficient Computation of Highly Oscillatory Integrals by Using QTT Tensor Approximation , 2014, Comput. Methods Appl. Math..

[278]  Fernando De la Torre,et al.  A Least-Squares Framework for Component Analysis , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[279]  G. Vidal.  Efficient classical simulation of slightly entangled quantum computations. , 2003, Physical review letters.

[280]  Andrzej Cichocki,et al.  Linked Component Analysis From Matrices to High-Order Tensors: Applications to Biomedical Data , 2015, Proceedings of the IEEE.

[281]  Moni Naor,et al.  Latent Variable Analysis and Signal Separation , 2015, Lecture Notes in Computer Science.

[282]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[283]  U. Schollwoeck.  The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[284]  B. Khoromskij.  O(dlog N)-Quantics Approximation of N-d Tensors in High-Dimensional Numerical Modeling , 2011 .

[285]  C. Lubich,et al.  A projector-splitting integrator for dynamical low-rank approximation , 2013, BIT Numerical Mathematics.

[286]  Denis Zorin,et al.  A Tensor-Train accelerated solver for integral equations in complex geometries , 2015, J. Comput. Phys..

[287]  Liqing Zhang,et al.  Bayesian Robust Tensor Factorization for Incomplete Multiway Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[288]  H. Vincent Poor,et al.  Robust iteratively reweighted Lasso for sparse tensor factorizations , 2014, 2014 IEEE Workshop on Statistical Signal Processing (SSP).

[289]  G. Evenbly,et al.  Entanglement Renormalization and Wavelets. , 2016, Physical review letters.

[290]  Andrzej Cichocki,et al.  Tensor Deflation for CANDECOMP/PARAFAC— Part I: Alternating Subspace Update Algorithm , 2015, IEEE Transactions on Signal Processing.

[291]  Steven R. White,et al.  Minimally entangled typical thermal state algorithms , 2010, 1002.1305.

[292]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part III: Alternating Least Squares Algorithms , 2008, SIAM J. Matrix Anal. Appl..

[293]  Brian C. Lovell,et al.  Extrinsic Methods for Coding and Dictionary Learning on Grassmann Manifolds , 2014, International Journal of Computer Vision.

[294]  Jennifer Seberry,et al.  The Strong Kronecker Product , 1994, J. Comb. Theory, Ser. A.

[295]  Kennedy,et al.  Rigorous results on valence-bond ground states in antiferromagnets. , 1987, Physical review letters.

[296]  Jure Leskovec,et al.  Tensor Spectral Clustering for Partitioning Higher-order Network Structures , 2015, SDM.

[297]  Radek Erban,et al.  Tensor methods for parameter estimation and bifurcation analysis of stochastic reaction networks , 2015, Journal of The Royal Society Interface.

[298]  Tamara G. Kolda,et al.  Parallel Tensor Compression for Large-Scale Scientific Data , 2015, 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[299]  Andrzej Cichocki,et al.  Fast Alternating LS Algorithms for High Order CANDECOMP/PARAFAC Tensor Factorizations , 2013, IEEE Transactions on Signal Processing.

[300]  Andrzej Cichocki,et al.  Tensor Decompositions: A New Concept in Brain Data Analysis? , 2013, ArXiv.

[301]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[302]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[303]  Eugene E. Tyrtyshnikov,et al.  Tucker Dimensionality Reduction of Three-Dimensional Arrays in Linear Time , 2008, SIAM J. Matrix Anal. Appl..

[304]  Lars Grasedyck,et al.  Variants of Alternating Least Squares Tensor Completion in the Tensor Train Format , 2015, SIAM J. Sci. Comput..

[305]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[306]  Lieven De Lathauwer,et al.  Optimization-Based Algorithms for Tensor Decompositions: Canonical Polyadic Decomposition, Decomposition in Rank-(Lr, Lr, 1) Terms, and a New Generalization , 2013, SIAM J. Optim..

[307]  Zenglin Xu,et al.  DinTucker: Scaling Up Gaussian Process Models on Large Multidimensional Arrays , 2016, AAAI.

[308]  Naotaka Fujii,et al.  Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[309]  Andrzej Cichocki,et al.  Extended HALS algorithm for nonnegative Tucker decomposition and its applications for multiway analysis and classification , 2011, Neurocomputing.

[310]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[311]  Daniele Venturi,et al.  Numerical methods for high-dimensional probability density function equations , 2016, J. Comput. Phys..

[312]  Pierre Comon,et al.  Tensors : A brief introduction , 2014, IEEE Signal Processing Magazine.

[313]  Narendra Ahuja,et al.  Out-of-core tensor approximation of multi-dimensional matrices of visual data , 2005, ACM Trans. Graph..

[314]  Sanjoy Dasgupta,et al.  Adaptive Control Processes , 2010, Encyclopedia of Machine Learning and Data Mining.

[315]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .