Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions

We propose a new method for low-rank approximation of Moore--Penrose pseudoinverses of large-scale matrices using tensor networks. The computed pseudoinverses can be useful for solving or preconditioning of large-scale overdetermined or underdetermined systems of linear equations. The computation is performed efficiently and stably based on the modified alternating least squares scheme using low-rank tensor train (TT) decompositions and tensor network contractions. The formulated large-scale optimization problem is reduced to sequential smaller-scale problems for which any standard and stable algorithms can be applied. A regularization technique is incorporated in order to alleviate ill-posedness and obtain robust low-rank approximations. Numerical simulation results illustrate that the regularized pseudoinverses of a wide class of nonsquare or nonsymmetric matrices admit good approximate low-rank TT representations. Moreover, we demonstrated that the computational cost of the proposed method is only loga...

[1]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[2]  Boris N. Khoromskij,et al.  Superfast Fourier Transform Using QTT Approximation , 2012 .

[3]  S. V. Dolgov,et al.  ALTERNATING MINIMAL ENERGY METHODS FOR LINEAR SYSTEMS IN HIGHER DIMENSIONS∗ , 2014 .

[4]  Grégory Legrain,et al.  Low-Rank Approximate Inverse for Preconditioning Tensor-Structured Linear Systems , 2013, SIAM J. Sci. Comput..

[5]  Dianne P. O'Leary,et al.  Optimal regularized low rank inverse approximation , 2015 .

[6]  Gene H. Golub,et al.  Matrix computations , 1983 .

[7]  Christine Tobler,et al.  Multilevel preconditioning and low‐rank tensor iteration for space–time simultaneous discretizations of parabolic PDEs , 2015, Numer. Linear Algebra Appl..

[8]  Andrzej Cichocki,et al.  Fundamental Tensor Operations for Large-Scale Data Analysis in Tensor Train Formats , 2014, ArXiv.

[9]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[10]  Jun-Feng Yin,et al.  Greville’s method for preconditioning least squares problems , 2011, Adv. Comput. Math..

[11]  Antonio Falcó,et al.  On Minimal Subspaces in Tensor Representations , 2012, Found. Comput. Math..

[12]  Nico Vervliet,et al.  Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis , 2014, IEEE Signal Processing Magazine.

[13]  Wolfgang Hackbusch Solution of linear systems in high spatial dimensions , 2015, Comput. Vis. Sci..

[14]  Ken Hayami,et al.  Generalized approximate inverse preconditioners for least squares problems , 2009 .

[15]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[16]  Johan A. K. Suykens,et al.  Learning with tensors: a framework based on convex optimization and spectral regularization , 2014, Machine Learning.

[17]  Andrzej Cichocki,et al.  Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions , 2014, ArXiv.

[18]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[19]  D. Savostyanov,et al.  Exact NMR simulation of protein-size spin systems using tensor train formalism , 2014, 1402.4516.

[20]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[21]  Andrzej Cichocki,et al.  Tensor Networks for Big Data Analytics and Large-Scale Optimization Problems , 2014, ArXiv.

[22]  Lieven De Lathauwer,et al.  A novel deterministic method for large-scale blind source separation , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[23]  Lieven De Lathauwer,et al.  Computation of tensor decompositions via numerical optimization , 2014 .

[24]  Dietrich Braess,et al.  Approximation of 1/x by exponential sums in [1, ∞) , 2005 .

[25]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[26]  B. Khoromskij Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[27]  Vladimir A. Kazeev,et al.  Direct Solution of the Chemical Master Equation Using Quantized Tensor Trains , 2014, PLoS Comput. Biol..

[28]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[29]  Christine Tobler,et al.  Low-rank tensor methods for linear systems and eigenvalue problems , 2012 .

[30]  J. Lasserre A trace inequality for matrix product , 1995, IEEE Trans. Autom. Control..

[31]  J. C. A. Barata,et al.  The Moore–Penrose Pseudoinverse: A Tutorial Review of the Theory , 2011, 1110.6882.

[32]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[33]  Boris N. Khoromskij,et al.  Tensor-Structured Preconditioners and Approximate Inverse of Elliptic Operators in ℝd , 2009 .

[34]  Yinchu Zhu,et al.  Breaking the curse of dimensionality in regression , 2017, ArXiv.

[35]  Lieven De Lathauwer,et al.  A survey on stochastic and deterministic tensorization for blind signal separation , 2015 .

[36]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[37]  P. Sonneveld,et al.  IDR(s): A family of simple and fast algorithms for solving large nonsymmetric linear systems , 2007 .

[38]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[39]  Daniel Kressner,et al.  Preconditioned Low-Rank Methods for High-Dimensional Elliptic PDE Eigenvalue Problems , 2011, Comput. Methods Appl. Math..

[40]  S. R. Simanca,et al.  On Circulant Matrices , 2012 .

[41]  L. Mirsky A trace inequality of John von Neumann , 1975 .

[42]  VLADIMIR A. KAZEEV,et al.  Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse , 2012, SIAM J. Matrix Anal. Appl..

[43]  Julianne Chung,et al.  An efficient approach for computing optimal low-rank regularized inverse matrices , 2014, ArXiv.

[44]  U. Schollwoeck The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[45]  B. Khoromskij O(dlog N)-Quantics Approximation of N-d Tensors in High-Dimensional Numerical Modeling , 2011 .

[46]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[47]  André Uschmajew,et al.  On Local Convergence of Alternating Schemes for Optimization of Convex Problems in the Tensor Train Format , 2013, SIAM J. Numer. Anal..

[48]  Lieven De Lathauwer,et al.  Exact line and plane search for tensor optimization , 2013, Computational optimization and applications.

[49]  Marcus J. Grote,et al.  Parallel Preconditioning with Sparse Approximate Inverses , 1997, SIAM J. Sci. Comput..

[50]  Daniel Kressner,et al.  Low-Rank Tensor Methods with Subspace Correction for Symmetric Eigenvalue Problems , 2014, SIAM J. Sci. Comput..

[51]  W. Hackbusch Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[52]  Yaoliang Yu,et al.  Rank/Norm Regularization with Closed-Form Solutions: Application to Subspace Clustering , 2011, UAI.

[53]  White,et al.  Density-matrix algorithms for quantum renormalization groups. , 1993, Physical review. B, Condensed matter.

[54]  Andrzej Cichocki,et al.  Very Large-Scale Singular Value Decomposition Using Tensor Train Networks , 2014, ArXiv.

[55]  M. Benzi,et al.  A comparative study of sparse approximate inverse preconditioners , 1999 .

[56]  Martin J. Mohlenkamp,et al.  Numerical operator calculus in higher dimensions , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[57]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[58]  Eugene E. Tyrtyshnikov,et al.  Tensor-Train Ranks for Matrices and Their Inverses , 2011, Comput. Methods Appl. Math..

[59]  S. Dolgov TT-GMRES: solution to a linear system in the structured tensor format , 2012, 1206.5512.

[60]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..