Stationary and nonstationary learning characteristics of the LMS adaptive filter

This paper describes the performance characteristics of the LMS adaptive filter, a digital filter composed of a tapped delay line and adjustable weights, whose impulse response is controlled by an adaptive algorithm. For stationary stochastic inputs, the mean-square error, the difference between the filter output and an externally supplied input called the "desired response," is a quadratic function of the weights, a paraboloid with a single fixed minimum point that can be sought by gradient techniques. The gradient estimation process is shown to introduce noise into the weight vector that is proportional to the speed of adaptation and number of weights. The effect of this noise is expressed in terms of a dimensionless quantity "misadjustment" that is a measure of the deviation from optimal Wiener performance. Analysis of a simple nonstationary case, in which the minimum point of the error surface is moving according to an assumed first-order Markov process, shows that an additional contribution to misadjustment arises from "lag" of the adaptive process in tracking the moving minimum point. This contribution, which is additive, is proportional to the number of weights but inversely proportional to the speed of adaptation. The sum of the misadjustments can be minimized by choosing the speed of adaptation to make equal the two contributions. It is further shown, in Appendix A, that for stationary inputs the LMS adaptive algorithm, based on the method of steepest descent, approaches the theoretical limit of efficiency in terms of misadjustment and speed of adaptation when the eigenvalues of the input correlation matrix are equal or close in value. When the eigenvalues are highly disparate (λmax/λmin> 10), an algorithm similar to LMS but based on Newton's method would approach this theoretical limit very closely.

[1]  G. TEMPLE,et al.  Relaxation Methods in Engineering Science , 1942, Nature.

[2]  Bernard Widrow Adaptive sampled-data systems , 1960 .

[3]  R. E. Kalman,et al.  Optimum Seeking Methods. , 1964 .

[4]  F. K. Becker,et al.  Automatic equalization for digital communication , 1965 .

[5]  F. K. Becker,et al.  Automatic equalization for digital communication , 1965 .

[6]  J. S. Koford,et al.  The use of an adaptive threshold element to design a linear optimal pattern classifier , 1966, IEEE Trans. Inf. Theory.

[7]  F. Smith Design of quasi-optimal minimum-time controllers , 1966 .

[8]  P. Lion Rapid identification of linear and nonlinear systems. , 1967 .

[9]  H. R. Rudin,et al.  An automatic equalizer for general-purpose communication channels , 1967 .

[10]  M. M. Sondhi,et al.  An adaptive echo canceller , 1967 .

[11]  B. Widrow,et al.  Adaptive antenna systems , 1967 .

[12]  R W Lucky,et al.  Principles of data communication , 1968 .

[13]  A. Gersho Adaptive equalization of highly dispersive channels for data transmission , 1969 .

[14]  L. J. Griffiths,et al.  A simple adaptive algorithm for real-time processing in antenna arrays , 1969 .

[15]  F. Downton Stochastic Approximation , 1969, Nature.

[16]  M. W. Sage,et al.  Recursive generalised-least-squares procedure for online identification of process parameters , 1969 .

[17]  G. M. Lance,et al.  An approximate steepest descent method for parameter identification , 1969 .

[18]  Lee D. Davisson Steady-state error in adaptive mean-square minimization , 1970, IEEE Trans. Inf. Theory.

[19]  T. Daniell,et al.  Adaptation in nonstationary applications , 1970 .

[20]  Bernard Widrow Adaptive Model Control Applied to Real-Time Blood-Pressure Regulation , 1971 .

[21]  K. Suryanarayanan,et al.  A unified approach to discrete-time systems identification† , 1971 .

[22]  R. E. Kalman,et al.  Aspects of network and system theory , 1971 .

[23]  Mischa Schwartz,et al.  A Rapidly Converging First-Order Training Algorithm for an Adaptive Equalizer , 1971, IEEE Trans. Inf. Theory.

[24]  O. L. Frost,et al.  An algorithm for linearly constrained adaptive array processing , 1972 .

[25]  J. Mendel,et al.  Discrete Techniques of Parameter Estimation: The Equation Error Formulation , 1973 .

[26]  J. Mendel,et al.  Discrete Techniques of Parameter Estimation: The Equation Error Formulation , 1973 .

[27]  L. Griffiths Rapid measurement of digital instantaneous frequency , 1975 .

[28]  J. Makhoul,et al.  Linear prediction: A tutorial review , 1975, Proceedings of the IEEE.

[29]  K. H. Mueller A new, fast-converging mean-square algorithm for adaptive equalizers with partial-response signaling , 1975, The Bell System Technical Journal.

[30]  B. Widrow,et al.  Adaptive noise cancelling: Principles and applications , 1975 .

[31]  W.F. Gabriel,et al.  Adaptive arrays—An introduction , 1976, Proceedings of the IEEE.

[32]  S. Applebaum,et al.  Adaptive arrays , 1976 .