Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis

The widespread use of multisensor technology and the emergence of big data sets have highlighted the limitations of standard flat-view matrix models and the necessity to move toward more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift toward models that are essentially polynomial, the uniqueness of which, unlike the matrix methods, is guaranteed under very mild and natural conditions. Benefiting from the power of multilinear algebra as their mathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints which match data properties and extract more general latent components in the data than matrix-based methods.

[1]  S. Goreinov,et al.  On cross approximation of multi-index arrays , 2008 .

[2]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[3]  Lieven De Lathauwer,et al.  Block Component Analysis, a New Concept for Blind Source Separation , 2012, LVA/ICA.

[4]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[5]  Nikos D. Sidiropoulos,et al.  Blind PARAFAC receivers for DS-CDMA systems , 2000, IEEE Trans. Signal Process..

[6]  F. L. Hitchcock Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[7]  J. Kruskal Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[8]  Anastasios Kyrillidis,et al.  Multi-Way Compressed Sensing for Sparse Low-Rank Tensors , 2012, IEEE Signal Processing Letters.

[9]  Vince D. Calhoun,et al.  A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data , 2009, NeuroImage.

[10]  P. Kroonenberg Applied Multiway Data Analysis , 2008 .

[11]  Pierre Comon,et al.  Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix , 2012, SIAM J. Matrix Anal. Appl..

[12]  Andrzej Cichocki,et al.  Canonical Polyadic Decomposition Based on a Single Mode Blind Source Separation , 2012, IEEE Signal Processing Letters.

[13]  Johan Håstad,et al.  Tensor Rank is NP-Complete , 1989, ICALP.

[14]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[15]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[16]  Daniel Kressner,et al.  Algorithm 941 , 2014 .

[17]  Lieven De Lathauwer,et al.  Blind Separation of Exponential Polynomials and the Decomposition of a Tensor in Rank-(Lr, Lr, 1) Terms , 2011, SIAM J. Matrix Anal. Appl..

[18]  Richard G. Baraniuk,et al.  Kronecker Compressive Sensing , 2012, IEEE Transactions on Image Processing.

[19]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[20]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[21]  Pierre Comon,et al.  Tensors : A brief introduction , 2014, IEEE Signal Processing Magazine.

[22]  Nikos D. Sidiropoulos,et al.  Robust iterative fitting of multilinear models , 2005, IEEE Transactions on Signal Processing.

[23]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[24]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[25]  Bart Vandereycken,et al.  The geometry of algorithms using hierarchical tensors , 2013, Linear Algebra and its Applications.

[26]  Fumikazu Miwakeichi,et al.  Decomposing EEG data into space–time–frequency components using Parallel Factor Analysis , 2004, NeuroImage.

[27]  Rasmus Bro,et al.  Multi-way Analysis with Applications in the Chemical Sciences , 2004 .

[28]  Andrzej Cichocki,et al.  Multidimensional compressed sensing and their applications , 2013, WIREs Data Mining Knowl. Discov..

[29]  H. Kiers,et al.  Selecting among three-mode principal component models of different types and complexities: a numerical convex hull based method. , 2006, The British journal of mathematical and statistical psychology.

[30]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[31]  C. L. Nikias,et al.  Higher-order spectra analysis : a nonlinear signal processing framework , 1993 .

[32]  W. Hackbusch Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[33]  Andrzej Cichocki,et al.  PARAFAC algorithms for large-scale problems , 2011, Neurocomputing.

[34]  Eugene E. Tyrtyshnikov,et al.  Tucker Dimensionality Reduction of Three-Dimensional Arrays in Linear Time , 2008, SIAM J. Matrix Anal. Appl..

[35]  Thomas Kailath,et al.  ESPRIT-estimation of signal parameters via rotational invariance techniques , 1989, IEEE Trans. Acoust. Speech Signal Process..

[36]  Lieven De Lathauwer,et al.  Optimization-Based Algorithms for Tensor Decompositions: Canonical Polyadic Decomposition, Decomposition in Rank-(Lr, Lr, 1) Terms, and a New Generalization , 2013, SIAM J. Optim..

[37]  D. Foster,et al.  Frequency of metamerism in natural scenes. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[38]  Naotaka Fujii,et al.  Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Nikos D. Sidiropoulos,et al.  Generalizing Carathéodory's uniqueness of harmonic parameterization to N dimensions , 2001, IEEE Trans. Inf. Theory.

[40]  Nikos D. Sidiropoulos,et al.  Kruskal's permutation lemma and the identification of CANDECOMP/PARAFAC and bilinear models with constant modulus constraints , 2004, IEEE Transactions on Signal Processing.

[41]  Nikos D. Sidiropoulos,et al.  Cramer-Rao lower bounds for low-rank decomposition of multidimensional arrays , 2001, IEEE Trans. Signal Process..

[42]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[43]  PROCEssIng magazInE IEEE Signal Processing Magazine , 2004 .

[44]  Andrzej Cichocki,et al.  Computing Sparse Representations of Multidimensional Signals Using Kronecker Bases , 2013, Neural Computation.

[45]  Morten Mørup,et al.  Applications of tensor (multiway array) factorizations and decompositions in data mining , 2011, WIREs Data Mining Knowl. Discov..

[46]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[47]  Demetri Terzopoulos,et al.  Multilinear Analysis of Image Ensembles: TensorFaces , 2002, ECCV.

[48]  Visa Koivunen,et al.  Sequential Unfolding SVD for Tensors With Applications in Array Signal Processing , 2009, IEEE Transactions on Signal Processing.

[49]  R. Bro Multiway calibration. Multilinear PLS , 1996 .

[50]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[51]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[52]  Pierre Comon,et al.  Nonnegative approximations of nonnegative tensors , 2009, ArXiv.

[53]  Andrzej Cichocki,et al.  Fast and unique Tucker decompositions via multiway blind source separation , 2012 .

[54]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[55]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[56]  Petros Drineas,et al.  Tensor-CUR decompositions for tensor-based data , 2006, KDD '06.

[57]  Bülent Yener,et al.  Unsupervised Multiway Data Analysis: A Literature Survey , 2009, IEEE Transactions on Knowledge and Data Engineering.

[58]  Nikos D. Sidiropoulos,et al.  Parallel factor analysis in sensor array processing , 2000, IEEE Trans. Signal Process..

[59]  Martin J. Mohlenkamp Musings on multilinear fitting , 2013 .

[60]  Severnyi Kavkaz Pseudo-Skeleton Approximations by Matrices of Maximal Volume , 2022 .

[61]  Arogyaswami Paulraj,et al.  An analytical constant modulus algorithm , 1996, IEEE Trans. Signal Process..

[62]  P. Laguna,et al.  Signal Processing , 2002, Yearbook of Medical Informatics.

[63]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[64]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[65]  Florian Roemer,et al.  Higher-Order SVD-Based Subspace Estimation to Improve the Parameter Estimation Accuracy in Multidimensional Harmonic Retrieval Problems , 2008, IEEE Transactions on Signal Processing.

[66]  Lieven De Lathauwer,et al.  On the Uniqueness of the Canonical Polyadic Decomposition of Third-Order Tensors - Part I: Basic Results and Uniqueness of One Factor Matrix , 2013, SIAM J. Matrix Anal. Appl..

[67]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[68]  A. Cichocki,et al.  Generalizing the column–row matrix decomposition to multi-way arrays , 2010 .

[69]  Renato Pajarola,et al.  TAMRESH – Tensor Approximation Multiresolution Hierarchy for Interactive Volume Visualization , 2013, Comput. Graph. Forum.

[70]  L. Lathauwer,et al.  On the Best Rank-1 and Rank-( , 2004 .

[71]  S. Goreinov,et al.  Pseudo-skeleton approximations by matrices of maximal volume , 1997 .

[72]  Zhi-Quan Luo,et al.  A Unified Convergence Analysis of Block Successive Minimization Methods for Nonsmooth Optimization , 2012, SIAM J. Optim..

[73]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[74]  Gordon Wetzstein,et al.  Tensor displays , 2012, SIGGRAPH '12.

[75]  B. Khoromskij Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[76]  André Uschmajew,et al.  Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation , 2012, SIAM J. Matrix Anal. Appl..

[77]  A. Stegeman On uniqueness conditions for Candecomp/Parafac and Indscal with full column rank in one mode , 2009 .

[78]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[79]  Zenglin Xu,et al.  Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis , 2011, ICML.

[80]  L. K. Hansen,et al.  Automatic relevance determination for multi‐way models , 2009 .

[81]  Johan A. K. Suykens,et al.  Learning with tensors: a framework based on convex optimization and spectral regularization , 2014, Machine Learning.

[82]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[83]  Nikos D. Sidiropoulos,et al.  Adaptive Algorithms to Track the PARAFAC Decomposition of a Third-Order Tensor , 2009, IEEE Transactions on Signal Processing.

[84]  Andrzej Cichocki,et al.  Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions , 2014, ArXiv.

[85]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part I: Lemmas for Partitioned Matrices , 2008, SIAM J. Matrix Anal. Appl..

[86]  Tamara G. Kolda,et al.  All-at-once Optimization for Coupled Matrix and Tensor Factorizations , 2011, ArXiv.

[87]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[88]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[89]  Shuzhong Zhang,et al.  Maximum Block Improvement and Polynomial Optimization , 2012, SIAM J. Optim..

[90]  Vince D. Calhoun,et al.  Joint Blind Source Separation by Multiset Canonical Correlation Analysis , 2009, IEEE Transactions on Signal Processing.

[91]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[92]  Lieven De Lathauwer,et al.  Canonical Polyadic Decomposition of Third-Order Tensors: Reduction to Generalized Eigenvalue Decomposition , 2013, SIAM J. Matrix Anal. Appl..

[93]  Michael Elad,et al.  Shape from moments - an estimation theory perspective , 2004, IEEE Transactions on Signal Processing.

[94]  Lieven De Lathauwer,et al.  Blind Signal Separation via Tensor Decomposition With Vandermonde Factor: Canonical Polyadic Decomposition , 2013, IEEE Transactions on Signal Processing.

[95]  Martin J. Mohlenkamp,et al.  Algorithms for Numerical Analysis in High Dimensions , 2005, SIAM J. Sci. Comput..

[96]  P. Paatero The Multilinear Engine—A Table-Driven, Least Squares Program for Solving Multilinear Problems, Including the n-Way Parallel Factor Analysis Model , 1999 .

[97]  A. Cichocki,et al.  Tensor decompositions for feature extraction and classification of high dimensional datasets , 2010 .

[98]  H. Kiers,et al.  Three-mode principal components analysis: choosing the numbers of components and sensitivity to local optima. , 2000, The British journal of mathematical and statistical psychology.

[99]  Zbynek Koldovský,et al.  Cramér-Rao-Induced Bounds for CANDECOMP/PARAFAC Tensor Decomposition , 2012, IEEE Transactions on Signal Processing.

[100]  B. Kowalski,et al.  Tensorial resolution: A direct trilinear decomposition , 1990 .

[101]  D. L. Donoho,et al.  Compressed sensing , 2006, IEEE Trans. Inf. Theory.

[102]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[103]  A. Cichocki Generalized Component Analysis and Blind Source Separation Methods for Analyzing Multichannel Brain Signals , 2006 .

[104]  Tamara G. Kolda,et al.  Scalable Tensor Factorizations for Incomplete Data , 2010, ArXiv.

[105]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[106]  Aapo Hyvärinen,et al.  Independent component analysis: recent advances , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[107]  Haiping Lu,et al.  A survey of multilinear subspace learning for tensor data , 2011, Pattern Recognit..

[108]  L. Lathauwer,et al.  Canonical Polyadic Decomposition with Orthogonality Constraints , 2012 .

[109]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[110]  T. Blumensath,et al.  Theory and Applications , 2011 .

[111]  Ali Taylan Cemgil,et al.  Probabilistic Latent Tensor Factorization , 2010, LVA/ICA.

[112]  R. Cattell “Parallel proportional profiles” and other principles for determining the choice of factors by rotation , 1944 .

[113]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[114]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[115]  B. Recht,et al.  Tensor completion and low-n-rank tensor recovery via convex optimization , 2011 .

[116]  J. Landsberg Tensors: Geometry and Applications , 2011 .

[117]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[118]  A. Stegeman,et al.  On the Non-Existence of Optimal Solutions and the Occurrence of “Degeneracy” in the CANDECOMP/PARAFAC Model , 2008, Psychometrika.

[119]  Yuan Qi,et al.  DinTucker: Scaling up Gaussian process models on multidimensional arrays with billions of elements , 2013, ArXiv.

[120]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[121]  P. Bahr,et al.  Sampling: Theory and Applications , 2020, Applied and Numerical Harmonic Analysis.