Sparsistent Learning of Varying-coefficient Models with Structural Changes

To estimate the changing structure of a varying-coefficient varying-structure (VCVS) model remains an important and open problem in dynamic system modelling, which includes learning trajectories of stock prices, or uncovering the topology of an evolving gene network. In this paper, we investigate sparsistent learning of a sub-family of this model — piecewise constant VCVS models. We analyze two main issues in this problem: inferring time points where structural changes occur and estimating model structure (i.e., model selection) on each of the constant segments. We propose a two-stage adaptive procedure, which first identifies jump points of structural changes and then identifies relevant covariates to a response on each of the segments. We provide an asymptotic analysis of the procedure, showing that with the increasing sample size, number of structural changes, and number of variables, the true model can be consistently selected. We demonstrate the performance of the method on synthetic data and apply it to the brain computer interface dataset. We also consider how this applies to structure estimation of time-varying probabilistic graphical models.

[1]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[2]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[3]  Martin J. Wainwright,et al.  Sharp thresholds for high-dimensional and noisy recovery of sparsity , 2006, ArXiv.

[4]  S. Geer,et al.  Locally adaptive regression splines , 1997 .

[5]  Jianqing Fan,et al.  Statistical Estimation in Varying-Coefficient Models , 1999 .

[6]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[7]  Emilie Lebarbier,et al.  Detecting multiple change-points in the mean of Gaussian process by model selection , 2005, Signal Process..

[8]  Zaïd Harchaoui,et al.  Kernel Change-point Analysis , 2008, NIPS.

[9]  O. Linton.  Local Regression Models , 2010 .

[10]  Klaus-Robert Müller,et al.  Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms , 2004, IEEE Transactions on Biomedical Engineering.

[11]  Z. Q. John Lu,et al.  Nonlinear Time Series: Nonparametric and Parametric Methods , 2004, Technometrics.

[12]  P. Perron,et al.  Estimating and testing linear models with multiple structural changes , 1995 .

[13]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[14]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[15]  R. Tibshirani,et al.  Varying‐Coefficient Models , 1993 .

[16]  S. Geer,et al.  On the conditions used to prove oracle results for the Lasso , 2009, 0910.0722.

[17]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[18]  Francis R. Bach,et al.  Bolasso: model consistent Lasso estimation through the bootstrap , 2008, ICML '08.

[19]  Zaïd Harchaoui,et al.  Catching Change-points with Lasso , 2007, NIPS.

[20]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Maximum Likelihood Estimation , 2007, ArXiv.

[21]  Le Song,et al.  Estimating time-varying networks , 2008, ISMB 2008.

[22]  Yingcun Xia,et al.  Shrinkage Estimation of the Varying Coefficient Model , 2008 .

[23]  Amr Ahmed,et al.  Recovering time-varying networks of dependencies in social and biological studies , 2009, Proceedings of the National Academy of Sciences.

[24]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[25]  A. Rinaldo.  Properties and refinements of the fused lasso , 2008, 0805.0234.

[26]  Chris H. Q. Ding,et al.  Spectral Relaxation for K-means Clustering , 2001, NIPS.

[27]  Le Song,et al.  KELLER: estimating time-varying interactions between genes , 2009, Bioinform..

[28]  N. Meinshausen,et al.  Stability selection , 2008, 0809.2932.

[29]  É. Moulines,et al.  Least‐squares Estimation of an Unknown Number of Shifts in a Time Series , 2000 .

[30]  F. Bunea.  Honest variable selection in linear and logistic regression models via $\ell_1$ and $\ell_1+\ell_2$ penalization , 2008, 0808.4051.

[31]  P. Perron,et al.  Computation and Analysis of Multiple Structural-Change Models , 1998 .