Sensitivity Analysis Using It[o-circumflex]--Malliavin Calculus and Martingales, and Application to Stochastic Optimal Control

We consider a multidimensional diffusion process $(X^\alpha_t)_{0\leq t\leq T}$ whose dynamics depends on a parameter $\alpha$. Our first purpose is to write as an expectation the sensitivity $\nabla_\alpha J(\alpha)$ for the expected cost $J(\alpha)=\mathbb{E}(f(X^\alpha_T))$, in order to evaluate it using Monte Carlo simulations. This issue arises, for example, from stochastic control problems (where the controller is parameterized, which reduces the control problem to a parametric optimization one) or from model misspecifications in finance. Previous evaluations of $\nabla_\alpha J(\alpha)$ using simulations were limited to smooth cost functions $f$ or to diffusion coefficients not depending on $\alpha$ (see Yang and Kushner, SIAM J. Control Optim., 29 (1991), pp. 1216--1249). In this paper, we cover the general case, deriving three new approaches to evaluate $\nabla_\alpha J(\alpha)$, which we call the Malliavin calculus approach, the adjoint approach, and the martingale approach. To accomplish this, we leverage Ito calculus, Malliavin calculus, and martingale arguments. In the second part of this work, we provide discretization procedures to simulate the relevant random variables; then we analyze their respective errors. This analysis proves that the discretization error is essentially linear with respect to the time step. This result, which was already known in some specific situations, appears to be true in this much wider context. Finally, we provide numerical experiments in random mechanics and finance and compare the different methods in terms of variance, complexity, computational time, and time discretization error.

[1]  S. R. Searle Linear Models , 1971 .

[2]  D. Gilbarg,et al.  Elliptic Partial Differential Equa-tions of Second Order , 1977 .

[3]  P. Kloeden,et al.  Numerical Solution of Stochastic Differential Equations , 1992 .

[4]  A. Friedman Partial Differential Equations of Parabolic Type , 1983 .

[5]  J. Bismut Large Deviations and the Malliavin Calculus , 1984 .

[6]  H. Kunita Stochastic differential equations and stochastic flows of diffeomorphisms , 1984 .

[7]  D. Stroock,et al.  Applications of the Malliavin calculus. II , 1985 .

[8]  Peter W. Glynn,et al.  Stochastic approximation for Monte Carlo optimization , 1986, WSC '86.

[9]  Alan Weiss,et al.  Sensitivity analysis via likelihood ratios , 1986, WSC '86.

[10]  Peter W. Glynn,et al.  Likelilood ratio gradient estimation: an overview , 1987, WSC '87.

[11]  A. Bensoussan Perturbation Methods in Optimal Control , 1988 .

[12]  P. Krée,et al.  Mathematics of random phenomena: Random vibrations of mechanical structures , 2012 .

[13]  S. Peng A general stochastic maximum principle for optimal control problems , 1990 .

[14]  D. Talay,et al.  Expansion of the global error for numerical schemes solving stochastic differential equations , 1990 .

[15]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[16]  P. Protter Stochastic integration and differential equations , 1990 .

[17]  H. Kushner,et al.  A Monte Carlo method for sensitivity analysis and parametric optimization of nonlinear stochastic systems , 1991 .

[18]  P. Glasserman,et al.  Some Guidelines and Guarantees for Common Random Numbers , 1992 .

[19]  Pierre L'Ecuyer,et al.  On the Convergence Rates of IPA and FDC Derivative Estimators , 1990, Oper. Res..

[20]  D. Nualart The Malliavin Calculus and Related Topics , 1995 .

[21]  D. Talay,et al.  The law of the Euler scheme for stochastic differential equations , 1996 .

[22]  P. Glasserman,et al.  Estimating security price derivatives using simulation , 1996 .

[23]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[24]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[25]  Harold J. Kushner,et al.  Stochastic Approximation Algorithms and Applications , 1997, Applications of Mathematics.

[26]  Anton Thalmaier On the differentiation of heat semigroups and Poisson integrals , 1997 .

[27]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[28]  Jakša Cvitanić,et al.  On Dynamic Measures of Risk , 1999 .

[29]  D. Nualart Analysis on Wiener space and anticipating stochastic calculus , 1998 .

[30]  P. Protter,et al.  Asymptotic error distributions for the Euler method for stochastic differential equations , 1998 .

[31]  Vivek S. Borkar,et al.  Actor-Critic - Type Learning Algorithms for Markov Decision Processes , 1999, SIAM J. Control. Optim..

[32]  X. Zhou,et al.  Stochastic Controls: Hamiltonian Systems and HJB Equations , 1999 .

[33]  H. Kushner Numerical Methods for Stochastic Control Problems in Continuous Time , 2000 .

[34]  E. Gobet Weak approximation of killed diffusion using Euler schemes , 2000 .

[35]  K. Elworthy,et al.  On the Geometry of Diffusion Operators and Stochastic Flows , 2000 .

[36]  E. Gobet Euler schemes and half-space approximation for the simulation of diffusion in a domain , 2001 .

[37]  Pierre-Louis Lions,et al.  Applications of Malliavin calculus to Monte-Carlo methods in finance. II , 2001, Finance Stochastics.

[38]  E. Gobet Local asymptotic mixed normality property for elliptic diffusion: a Malliavin calculus approach , 2001 .

[39]  Arturo Kohatsu-Higa,et al.  Weak Approximations. A Malliavin Calculus Approach , 1999, Math. Comput..

[40]  J. Yong,et al.  Stochastic Linear Quadratic Optimal Control Problems , 2001 .

[41]  J. Picard Gradient estimates for some diffusion semigroups , 2002 .

[42]  Arturo Kohatsu-Higa,et al.  Variance Reduction Methods for Simulation of Densities on Wiener Space , 2002, SIAM J. Numer. Anal..

[43]  E. Gobet LAN property for ergodic diffusions with discrete observations , 2002 .

[44]  P. Cattiaux,et al.  Hypoelliptic non-homogeneous diffusions , 2002 .

[45]  Wolfgang J. Runggaldier,et al.  On Stochastic Control in Finance , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[46]  D. Talay,et al.  Approximation of quantiles of components of diffusion processes , 2004 .

[47]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.