Error bound analysis of the least-mean-squares algorithm in linear models

Time-varying formulation is pervasive in dynamic control systems, where tracking parameters characterizing dynamic properties is central task. General conditions for exponential stability of such systems have been fully discussed in prior work of others. In this paper we build practical (computable) error bound analysis of the stochastic gradient algorithm when the loss function is time-dependent and quadratic in the parameters, as arising from standard linear regression model. The long term goal is to address this problem in general nonlinear models. This paper is the first step towards this aim.

[1]  L. Gerencsér On a class of mixing processes , 1989 .

[2]  L. Gerencsér Rate of convergence of the LMS method , 1995 .

[3]  Lennart Ljung,et al.  Performance analysis of general tracking algorithms , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[4]  G. Pflug,et al.  Stochastic approximation and optimization of random systems , 1992 .

[5]  eon BottouAT Stochastic Gradient Learning in Neural Networks , 2022 .

[6]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[7]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[8]  G. Wittum,et al.  Adaptive filtering , 1997 .

[9]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[10]  Andrew H. Jazwinski,et al.  Adaptive filtering , 1969, Autom..

[11]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[12]  Léon Bottou,et al.  Stochastic Learning , 2003, Advanced Lectures on Machine Learning.

[13]  D. Bernstein Matrix Mathematics: Theory, Facts, and Formulas , 2009 .

[14]  E. Eweda,et al.  Second-order convergence analysis of stochastic adaptive linear filtering , 1983 .

[15]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.