Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method

A new generalized multilinear regression model, termed the higher order partial least squares (HOPLS), is introduced with the aim to predict a tensor (multiway array) Y from a tensor X through projecting the data onto the latent space and performing regression on the corresponding latent variables. HOPLS differs substantially from other regression models in that it explains the data by a sum of orthogonal Tucker tensors, while the number of orthogonal loadings serves as a parameter to control model complexity and prevent overfitting. The low-dimensional latent space is optimized sequentially via a deflation operation, yielding the best joint subspace approximation for both X and Y. Instead of decomposing X and Y individually, higher order singular value decomposition on a newly defined generalized cross-covariance tensor is employed to optimize the orthogonal loadings. A systematic comparison on both synthetic data and real-world decoding of 3D movement trajectories from electrocorticogram signals demonstrate the advantages of HOPLS over the existing methods in terms of better predictive ability, suitability to handle small sample sizes, and robustness to noise.

[1]  Anthony Randal McIntosh,et al.  Partial Least Squares (PLS) methods for neuroimaging: A tutorial and review , 2011, NeuroImage.

[2]  Rasmus Bro,et al.  MULTI-WAY ANALYSIS IN THE FOOD INDUSTRY Models, Algorithms & Applications , 1998 .

[3]  Herman Wold,et al.  Soft modelling: The Basic Design and Some Extensions , 1982 .

[4]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[5]  Yasuo Nagasaka,et al.  Multidimensional Recording (MDR) and Data Sharing: An Ecological Open Research and Educational Platform for Neuroscience , 2011, PloS one.

[6]  Anthony Randal McIntosh,et al.  Partial least squares analysis of neuroimaging data: applications and advances , 2004, NeuroImage.

[7]  S. Wold,et al.  Orthogonal projections to latent structures (O‐PLS) , 2002 .

[8]  Jammalamadaka Introduction to Linear Regression Analysis (3rd ed.) , 2003 .

[9]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[10]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[11]  Age K. Smilde,et al.  Multiway covariates regression models , 1999 .

[12]  Rolf Ergon,et al.  PLS score–loading correspondence and a bi‐orthogonal factorization , 2002 .

[13]  C. Mehring,et al.  Encoding of Movement Direction in Different Frequency Ranges of Motor Cortical Local Field Potentials , 2005, The Journal of Neuroscience.

[14]  Rasmus Bro,et al.  On the difference between low-rank and subspace approximation: improved model for multi-linear PLS regression , 2001 .

[15]  Natasa Kovacevic,et al.  Groupwise independent component decomposition of EEG data and partial least square analysis , 2007, NeuroImage.

[16]  E. Acar,et al.  Seizure Recognition on Epilepsy Feature Tensor , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  S. de Jong,et al.  Multiway calibration in 3D QSAR , 1997 .

[18]  Age K. Smilde,et al.  Comments on multilinear PLS , 1997 .

[19]  L.J. Trejo,et al.  Brain-computer interfaces for 1-D and 2-D cursor control: designs using volitional control of the EEG spectrum or steady-state visual evoked potentials , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  R. Bro Multiway calibration. Multilinear PLS , 1996 .

[21]  J. Brian Gray,et al.  Introduction to Linear Regression Analysis , 2002, Technometrics.

[22]  H. Wold Soft Modelling by Latent Variables: The Non-Linear Iterative Partial Least Squares (NIPALS) Approach , 1975, Journal of Applied Probability.

[23]  Kimito Funatsu,et al.  Rational choice of bioactive conformations through use of conformation analysis and 3-way partial least squares modeling , 2000 .

[24]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part II: Definitions and Uniqueness , 2008, SIAM J. Matrix Anal. Appl..

[25]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[26]  Elaine B. Martin,et al.  Model selection for partial least squares regression , 2002 .

[27]  Fumikazu Miwakeichi,et al.  Concurrent EEG/fMRI analysis by multiway Partial Least Squares , 2004, NeuroImage.

[28]  Rasmus Bro,et al.  Standard error of prediction for multilinear PLS 2. Practical implementation in fluorescence spectroscopy , 2005 .

[29]  John Shawe-Taylor,et al.  Efficient Sparse Kernel Feature Extraction Based on Partial Least Squares , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Sijmen de Jong,et al.  Multiway calibration in 3D QSAR , 1997 .

[31]  R. Bro Review on Multiway Analysis in Chemistry—2000–2005 , 2006 .

[32]  S. Wold,et al.  PLS-regression: a basic tool of chemometrics , 2001 .

[33]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[34]  T. Kolda Multilinear operators for higher-order decompositions , 2006 .

[35]  VandewalleJoos,et al.  On the Best Rank-1 and Rank-(R1,R2,. . .,RN) Approximation of Higher-Order Tensors , 2000 .

[36]  Hyunsoo Kim,et al.  A three-stage framework for gene expression data analysis by L1-norm support vector regression , 2005, Int. J. Bioinform. Res. Appl..

[37]  Rasmus Bro,et al.  Multi-way Analysis with Applications in the Chemical Sciences , 2004 .

[38]  Roman Rosipal,et al.  Overview and Recent Advances in Partial Least Squares , 2005, SLSFS.

[39]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[40]  S. Wold,et al.  The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses , 1984 .

[41]  Richard G. Brereton,et al.  Two-way, unfolded three-way and three-mode partial least squares calibration of diode array HPLC chromatograms for the quantitation of low-level pharmaceutical impurities , 1999 .

[42]  Rasmus Bro,et al.  Standard error of prediction for multilinear PLS , 2005 .

[43]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[44]  Tamara G. Kolda,et al.  Scalable Tensor Factorizations for Incomplete Data , 2010, ArXiv.

[45]  Stefan Schaal,et al.  Locally Weighted Projection Regression : An O(n) Algorithm for Incremental Real Time Learning in High Dimensional Space , 2000 .

[46]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[47]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[48]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[49]  H. Abdi Partial least squares regression and projection on latent structure regression (PLS Regression) , 2010 .

[50]  Roman Rosipal,et al.  Kernel Partial Least Squares Regression in Reproducing Kernel Hilbert Space , 2002, J. Mach. Learn. Res..

[51]  A. R. McIntosh,et al.  Spatiotemporal analysis of event-related fMRI data using partial least squares , 2004, NeuroImage.

[52]  Alejandro C. Olivieri,et al.  When unfolding is better: unique success of unfolded partial least-squares regression with residual bilinearization for the processing of spectral-pH data with strong spectral overlapping. Analysis of fluoroquinolones in human urine based on flow-injection pH-modulated synchronous fluorescence data , 2009, The Analyst.

[53]  Naotaka Fujii,et al.  Long-Term Asynchronous Decoding of Arm Motion Using Electrocorticographic Signals in Monkeys , 2009, Front. Neuroeng..