Generalized Cross-entropy Methods with Applications to Rare-event Simulation and Optimization

The cross-entropy and minimum cross-entropy methods are well-known Monte Carlo simulation techniques for rare-event probability estimation and optimization. In this paper, we investigate how these methods can be eXtended to provide a general non-parametric cross-entropy framework based on φ-divergence distance measures. We show how the χ 2 distance, in particular, yields a viable alternative to the Kullback—Leibler distance. The theory is illustrated with various eXamples from density estimation, rare-event simulation and continuous multi-eXtremal optimization.

[1]  J. Aitchison,et al.  Multivariate binary discrimination by the kernel method , 1976 .

[2]  D. M. Titterington,et al.  A Comparative Study of Kernel-Based Density Estimates for Categorical Data , 1980 .

[3]  A. Bowman.  A comparative study of some kernel-based nonparametric density estimators , 1985 .

[4]  H. K. Kesavan,et al.  The generalized maximum entropy principle , 1989, IEEE Trans. Syst. Man Cybern..

[5]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[6]  Jagat Narain Kapur,et al.  Measures of information and their applications , 1994 .

[7]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[8]  L. Devroye,et al.  Nonparametric density estimation : the L[1] view , 1987 .

[9]  J. Borwein,et al.  Duality relationships for entropy-like minimization problems , 1991 .

[10]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .

[11]  Jerry Nedelman,et al.  Book review: “Bayesian Data Analysis,” Second Edition by A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin Chapman & Hall/CRC, 2004 , 2005, Comput. Stat..

[12]  Hiremaglur K. Kesavan,et al.  The generalized maximum entropy principle (with applications) , 1987 .

[13]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[14]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[15]  Dirk P. Kroese,et al.  Cross‐Entropy Method , 2011 .

[16]  Jan Havrda,et al.  Quantification method of classification processes. Concept of structural a-entropy , 1967, Kybernetika.

[17]  R. Rubinstein.  A Stochastic Minimum Cross-Entropy Method for Combinatorial Optimization and Rare-event Estimation* , 2005 .

[18]  Claude Lemaréchal,et al.  Dual Methods in Entropy Maximization. Application to Some Problems in Crystallography , 1992, SIAM J. Optim..

[19]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[20]  Christian P. Robert,et al.  Monte Carlo Statistical Methods (Springer Texts in Statistics) , 2005 .

[21]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[22]  J A Anderson,et al.  A statistical aid to the diagnosis of keratoconjunctivitis sicca. , 1972, The Quarterly journal of medicine.

[23]  Zdravko I. Botev,et al.  Stochastic Methods for Optimization and Machine Learning , 2005 .

[24]  J. Simonoff.  Smoothing Methods in Statistics , 1998 .

[25]  P. Hall.  On nonparametric multivariate binary discrimination , 1981 .

[26]  J. Marron,et al.  Progress in data-based bandwidth selection for kernel density estimation , 1996 .

[27]  Marc Teboulle,et al.  Penalty Functions and Duality in Stochastic Programming Via ϕ-Divergence Functionals , 1987, Math. Oper. Res..

[28]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[29]  Luc Devroye,et al.  Nonparametric Density Estimation , 1985 .

[30]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[31]  B. Turlach,et al.  Reducing bias in curve estimation by use of weights , 1999 .

[32]  Roger Sauter,et al.  In All Likelihood , 2002, Technometrics.