Time Varying Regression with Hidden Linear Dynamics

We revisit a model for time-varying linear regression that assumes the unknown parameters evolve according to a linear dynamical system. Counterintuitively, we show that when the underlying dynamics are stable the parameters of this model can be estimated from data by combining just two ordinary least squares estimates. We offer a finite sample guarantee on the estimation error of our method and discuss certain advantages it has over ExpectationMaximization (EM), which is the main approach proposed by prior work.

[1]  竹安 数博,et al.  Time series analysis and its applications , 2007 .

[2]  George J. Pappas,et al.  Sample Complexity of Kalman Filtering for Unknown Systems , 2019, L4DC.

[3]  Martin J. Wainwright,et al.  Statistical guarantees for the EM algorithm: From population to sample-based analysis , 2014, ArXiv.

[4]  Leigh Tesfatsion,et al.  TIME-VARYING LINEAR REGRESSION VIA FLEXIBLE LEAST SQUARES† , 1989 .

[5]  A. Ullah,et al.  Expectation of quadratic forms in normal and nonnormal variables with applications , 2010 .

[6]  John L. Junkins,et al.  Time-Varying Eigensystem Realization Algorithm , 2010 .

[7]  R. Shumway,et al.  AN APPROACH TO TIME SERIES SMOOTHING AND FORECASTING USING THE EM ALGORITHM , 1982 .

[8]  Alf Isaksson.  Identification of Time Varying Systems Through Adaptive Kalman Filtering , 1987 .

[9]  G. Chow,et al.  Econometric analysis by control methods , 1983 .

[10]  Tengyu Ma,et al.  Gradient Descent Learns Linear Dynamical Systems , 2016, J. Mach. Learn. Res..

[11]  Wei Biao Wu,et al.  Inference of time-varying regression models , 2012, 1208.3552.

[12]  George J. Pappas,et al.  Finite Sample Analysis of Stochastic System Identification , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[13]  Michael I. Jordan,et al.  Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification , 2018, COLT.

[14]  Jan R. Magnus,et al.  The expectation of products of quadratic forms in normal variables: The practice Statistica Neerlandica , 1979 .

[15]  R. Douc,et al.  CONSISTENCY OF THE MAXIMUM LIKELIHOOD ESTIMATOR FOR GENERAL HIDDEN MARKOV MODELS , 2009, 0912.4480.

[16]  Munther A. Dahleh,et al.  Finite Time LTI System Identification , 2021, J. Mach. Learn. Res..

[17]  Samet Oymak,et al.  Revisiting Ho–Kalman-Based System Identification: Robustness and Finite-Sample Analysis , 2022, IEEE Transactions on Automatic Control.

[18]  Jules Depersin,et al.  A spectral algorithm for robust regression with subgaussian rates , 2020, ArXiv.

[19]  T. Lubik,et al.  Time-Varying Parameter Vector Autoregressions: Specification, Estimation, and an Application , 2016 .

[20]  G. A. Young,et al.  High‐dimensional Statistics: A Non‐asymptotic Viewpoint, Martin J.Wainwright, Cambridge University Press, 2019, xvii 552 pages, £57.99, hardback ISBN: 978‐1‐1084‐9802‐9 , 2020, International Statistical Review.

[21]  Mohammad Emtiyaz Khan,et al.  An Expectation-Maximization Algorithm Based Kalman Smoother Approach for Event-Related Desynchronization (ERD) Estimation from EEG , 2007, IEEE Transactions on Biomedical Engineering.

[22]  Kevin G. Jamieson,et al.  Active Learning for Identification of Linear Dynamical Systems , 2020, COLT.

[23]  Identification and Estimation of Time Varying Models with Structural Variability of the Parameters , 1983 .

[24]  Benjamin Recht,et al.  Non-Asymptotic Analysis of Robust Control from Coarse-Grained Identification , 2017, ArXiv.

[25]  Karan Singh,et al.  Learning Linear Dynamical Systems via Spectral Filtering , 2017, NIPS.

[26]  Prasad Raghavendra,et al.  Algorithms for heavy-tailed statistics: regression, covariance estimation, and beyond , 2019, STOC.

[27]  Max Simchowitz,et al.  Learning Linear Dynamical Systems with Semi-Parametric Least Squares , 2019, COLT.

[28]  Bassam Bamieh,et al.  Identification of linear parameter varying models , 1999, Proceedings of the 38th IEEE Conference on Decision and Control (Cat. No.99CH36304).

[29]  Jer-Nan Juang,et al.  An eigensystem realization algorithm for modal parameter identification and model reduction. [control systems design for large space structures] , 1985 .

[30]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[31]  Kefu Liu,et al.  IDENTIFICATION OF LINEAR TIME-VARYING SYSTEMS , 1997 .

[32]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[33]  Alexander Katsevich,et al.  Covariance Matrix Estimation for the Cryo-EM Heterogeneity Problem , 2013, SIAM J. Imaging Sci..

[34]  John S. White.  THE LIMITING DISTRIBUTION OF THE SERIAL CORRELATION COEFFICIENT IN THE EXPLOSIVE CASE , 1958 .

[35]  A S Azari,et al.  Modeling mortality fluctuations in Los Angeles as functions of pollution and weather effects. , 1988, Environmental research.

[36]  Alexander Rakhlin,et al.  Near optimal finite time identification of arbitrary linear dynamical systems , 2018, ICML.