Model-free control of nonlinear stochastic systems in discrete time

Consider the problem of developing a controller for general (nonlinear and stochastic) discrete-time systems, where the equations governing the system are unknown. This paper presents an approach based on estimating a controller without building or assuming a model for the system. Such an approach has potential advantages in, e.g. accommodating systems with time varying dynamics. The controller is constructed through use of a function approximator (FA) such as a neural network or polynomial (no FA is used for the unmodeled system equations). This involves the estimation of the unknown parameters within the FA. However, since no functional form is being assumed for the system equations, the gradient of the loss function for use in standard optimization algorithms is not available. Therefore, this paper considers the use of a stochastic approximation algorithm that is based on a simultaneous perturbation gradient approximation, which requires only system measurements (not a system model). Related to this, a convergence result for stochastic approximation algorithms with time-varying objective functions is established. It is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite-difference gradient approximations.

[1]  George N. Saridis,et al.  Self-organizing control of stochastic systems , 1977 .

[2]  T. Soderstrom,et al.  Stationary performance of linear stochastic systems under single step optimal control , 1982 .

[3]  Robert M. Sanner,et al.  Gaussian Networks for Direct Adaptive Control , 1991, 1991 American Control Conference.

[4]  J. Spall,et al.  Direct adaptive control of nonlinear systems using neural networks and stochastic approximation , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[5]  A. Benveniste,et al.  A measure of the tracking capability of recursive stochastic algorithms with constant gains , 1982 .

[6]  P.J. Antsaklis,et al.  Implementations of learning control systems using neural networks , 1992, IEEE Control Systems.

[7]  Tryphon T. Georgiou Aspects of Modeling, Identification and Control of Dynamical Systems. , 1995 .

[8]  Kurt Hornik,et al.  Convergence of learning algorithms with constant learning rates , 1991, IEEE Trans. Neural Networks.

[9]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[10]  J Thibault,et al.  On‐line prediction of fermentation variables using neural networks , 1990, Biotechnology and bioengineering.

[11]  Thomas Parisini,et al.  Neural approximations for multistage optimal control of nonlinear stochastic systems , 1996, IEEE Trans. Autom. Control..

[12]  James C. Spall,et al.  A neural network controller for systems with unmodeled dynamics with applications to wastewater treatment , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[13]  J. Spall,et al.  Nonlinear adaptive control using neural networks: estimation with a smoothed form of simultaneous perturbation gradient approximation , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[14]  Dejan J. Sobajic,et al.  Neural-net computing and the intelligent control of systems , 1992 .

[15]  E. Eweda,et al.  Second-order convergence analysis of stochastic adaptive linear filtering , 1983 .

[16]  J. Spall A second order stochastic approximation algorithm using only function measurements , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[17]  Kumpati S. Narendra,et al.  Neural networks in control systems , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[18]  K Y San,et al.  The design of controllers for batch bioreactors , 1988, Biotechnology and bioengineering.

[19]  D.A. Handelman,et al.  Theory and development of higher-order CMAC neural networks , 1992, IEEE Control Systems.

[20]  H. Kushner,et al.  Asymptotic Properties of Stochastic Approximations with Constant Coefficients. , 1981 .

[21]  Denis Dochain,et al.  Adaptive identification and control algorithms for nonlinear bacterial growth systems , 1984, Autom..

[22]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[23]  D. Bayard A forward method for optimal stochastic nonlinear and adaptive control , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[24]  F.-C. Chen,et al.  Back-propagation neural networks for nonlinear self-tuning adaptive control , 1990, IEEE Control Systems Magazine.

[25]  J. Spall Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[26]  D. C. Chin,et al.  A more efficient global optimization algorithm based on Styblinski and Tang , 1994, Neural Networks.

[27]  W. Rudin Principles of mathematical analysis , 1964 .