Variance Estimation For Online Regression via Spectrum Thresholding

We consider the dynamic linear regression problem, where the predictor vector may vary with time. This problem can be modeled as a linear dynamical system, where the parameters that need to be learned are the variance of both the process noise and the observation noise. While variance estimation for dynamic regression is a natural problem, with a variety of applications, existing approaches to this problem either lack guarantees or only have asymptotic guarantees without explicit rates. In addition, all existing approaches rely strongly on Guassianity of the noises. In this paper we study the global system operator: the operator that maps the noise vectors to the output. In particular, we obtain estimates on its spectrum, and as a result derive the first known variance estimators with finite sample complexity guarantees. Moreover, our results hold for arbitrary sub Gaussian distributions of noise terms. We evaluate the approach on synthetic and real-world benchmarks.

[1]  M. Kreĭn,et al.  Introduction to the theory of linear nonselfadjoint operators , 1969 .

[2]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[3]  Guanrong Chen,et al.  Kalman Filtering with Real-time Applications , 1987 .

[4]  Giovanni Petris,et al.  An R Package for Dynamic Linear Models , 2010 .

[5]  Pierre Pinson,et al.  Global Energy Forecasting Competition 2012 , 2014 .

[6]  Shie Mannor,et al.  On-Line Learning of Linear Dynamical Systems: Exponential Forgetting in Kalman Filters , 2018, AAAI.

[7]  Sham M. Kakade,et al.  A tail inequality for quadratic forms of subgaussian random vectors , 2011, ArXiv.

[8]  P. Bélanger.  Estimation of noise covariance matrices for a linear time-varying stochastic process , 1972, Autom..

[9]  Bart De Moor,et al.  Subspace Identification for Linear Systems: Theory ― Implementation ― Applications , 2011 .

[10]  R. Bhatia.  Matrix Analysis , 1996 .

[11]  Steven C. H. Hoi,et al.  Online ARIMA Algorithms for Time Series Prediction , 2016, AAAI.

[12]  Martin Zinkevich,et al.  Online Convex Programming and Generalized Infinitesimal Gradient Ascent , 2003, ICML.

[13]  Yi Zhang,et al.  Spectral Filtering for General Linear Dynamical Systems , 2018, NeurIPS.

[14]  Sysi-Aho Marko,et al.  The Finite Difference Method in Partial Differential Equations , 2001 .

[15]  Michael A. West,et al.  Bayesian forecasting and dynamic models (2nd ed.) , 1997 .

[16]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986, Encyclopedia of Big Data.

[17]  Shie Mannor,et al.  Online Learning for Time Series Prediction , 2013, COLT.

[18]  Yuanqing Xia,et al.  An adaptive Kalman filter estimating process noise covariance , 2017, Neurocomputing.

[19]  R. Mehra.  On the identification of variances and adaptive Kalman filtering , 1970 .

[20]  Ondrej Straka,et al.  Design of measurement difference autocovariance method for estimation of process and measurement noise covariances , 2018, Autom..

[21]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[22]  Elad Hazan,et al.  Online Learning of Linear Dynamical Systems , 2017, NIPS 2017.

[23]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[24]  F. T. Wright,et al.  A Bound on Tail Probabilities for Quadratic Forms in Independent Random Variables , 1971 .