Paper Information - Estimation and control in Markov chains

Estimation and control in Markov chains

We consider a finite controlled Markov chain, the description of which depends on an unknown parameter a, and investigate the following control policy. To each a an optimal stationary control is associated. a is estimated recurrently from the trajectory by the minimum contrast method, and the optimal stationary control corresponding to the estimate is used. We present asymptotic properties of the estimate and of the criterion function. They follow from the law of large numbers and from the central limit theorem for controlled Markov chains derived with the aid of martingales.

P. Mandl | P. Mandl | Petr Mandl

[1] R. Bellman. A Markovian Decision Process , 1957 .

[2] Ronald A. Howard,et al. Dynamic Programming and Markov Processes , 1960 .

[3] Patrick Billingsley,et al. Statistical inference for Markov processes , 1961 .

[4] P. Billingsley,et al. The Lindeberg-Lévy theorem for martingales , 1961 .

[5] Michel Loève,et al. Probability Theory I , 1977 .

[6] J. Pfanzagl. On the measurability and consistency of minimum contrast estimates , 1969 .

[7] B. M. Brown,et al. Martingale convergence to infinitely divisible laws with finite variances , 1971 .

[8] Petr Mandl. On the variance in controlled Markov chains , 1971, Kybernetika.

[9] B. M. Brown,et al. Martingale Central Limit Theorems , 1971 .

[10] P. Gänssler. Note on minimum contrast estimates forMarkov processes , 1972 .

[11] P. Mandl. An application of itô's formula to stochastic control systems , 1972 .

[12] P. Mandl. On the adaptive control of finite state Markov processes , 1973 .

[13] Petr Mandl. A connection between controlled Markov chains and martingales , 1973, Kybernetika.