Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization

We provide algorithms to compute the performance derivatives of Markov chains with respect to changes in their transition matrices and of Markov processes with respect to changes in their infinitesimal generators. Our algorithms are readily applicable to the control and optimization of these Markov systems, since they are based on analyzing a single sample path and do not need explicit specification of transition matrices, nor infinitesimal generators. Compared to the infinitesimal perturbation analysis, the algorithms have a wider scope of application and require nearly the same computational effort. Numerical examples are provided to illustrate the applications of the algorithms. In particular, we apply one of our algorithms to a closed queueing network and the results are promising.

[1]  J. Keilson.  Markov Chain Models--Rarity And Exponentiality , 1979 .

[2]  Christos G. Cassandras,et al.  Infinitesimal and finite perturbation analysis for queueing networks , 1982, 1982 21st IEEE Conference on Decision and Control.

[3]  Y. Ho,et al.  Smoothed (conditional) perturbation analysis of discrete event dynamical systems , 1987 .

[4]  Peter W. Glynn,et al.  Likelilood ratio gradient estimation: an overview , 1987, WSC '87.

[5]  R. Suri,et al.  Perturbation analysis gives strongly consistent sensitivity estimates for the M/G/ 1 queue , 1988 .

[6]  C. Cassandras,et al.  On-line sensitivity analysis of Markov chains , 1989 .

[7]  Alan Weiss,et al.  Sensitivity Analysis for Simulations via Likelihood Ratios , 1989, Oper. Res..

[8]  Paul Glasserman,et al.  Gradient Estimation Via Perturbation Analysis , 1990 .

[9]  Xi-Ren Cao,et al.  Perturbation analysis of discrete event dynamic systems , 1991 .

[10]  Pirooz Vakili,et al.  Using a standard clock technique for efficient simulation , 1991, Oper. Res. Lett..

[11]  Peter W. Glynn,et al.  Gradient estimation for ratios , 1991, 1991 Winter Simulation Conference Proceedings..

[12]  George Liberopoulos,et al.  Perturbation Analysis for the Design of Flexible Manufacturing System Flow Controllers , 1992, Oper. Res..

[13]  Pierre Brémaud,et al.  On the pathwise computation of derivatives with respect to the rate of a point process: The phantom RPA method , 1992, Queueing Syst. Theory Appl..

[14]  Pierre Brémaud,et al.  Maximal coupling and rare perturbation sensitivity analysis , 1992, Queueing Syst. Theory Appl..

[15]  M. Fu,et al.  Addendum to "Extensions and generalizations of smoothed perturbation analysis in a generalized semi-Markov process framework" , 1992 .

[16]  E. Chong,et al.  Optimization of queues using an infinitesimal perturbation analysis-based stochastic algorithm with general update times , 1993 .

[17]  Houmin Yan,et al.  Finding optimal number of Kanbans in a manufacturing system via stochastic approximation and perturbation analysis , 1994 .

[18]  Edwin K. P. Chong,et al.  Discrete event systems: Modeling and performance analysis , 1994, Discret. Event Dyn. Syst..

[19]  Christian van Delft,et al.  Convergence of stochastic approximation coupled with perturbation analysis in a class of manufacturing flow control models , 1994, Discret. Event Dyn. Syst..

[20]  Felix Wu,et al.  Dynamic bandwidth allocation using infinitesimal perturbation analysis , 1994, Proceedings of INFOCOM '94 Conference on Computer Communications.

[21]  Charles Leake,et al.  Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method , 1994 .

[22]  Jason H. Goodfriend,et al.  Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method , 1995 .

[23]  Y. Ho,et al.  Structural infinitesimal perturbation analysis (SIPA) for derivative estimation of discrete-event dynamic systems , 1995, IEEE Trans. Autom. Control..

[24]  Cao Xi-ren.  Uniformization and performance sensitivity estimation in closed queueing networks , 1996 .

[25]  L. Dai.  Sensitivity analysis of stationary performance measures for Markov chains , 1996 .

[26]  LIYI DAI,et al.  Rate of Convergence for Derivative Estimation of Discrete-Time Markov Chains via Finite-Difference Approximation with Common Random Numbers , 1997, SIAM J. Appl. Math..

[27]  Xi-Ren Cao,et al.  Perturbation realization, potentials, and sensitivity analysis of Markov processes , 1997, IEEE Trans. Autom. Control..

[28]  P. Glynn.  LIKELIHOOD RATIO GRADIENT ESTIMATION : AN OVERVIEW by , 2022 .