Theoretical Framework for Comparing Several Stochastic Optimization Approaches

This paper establishes a framework for formal comparisons of several leading optimization algorithms, establishing guidance to practitioners for when to use or not use a particular method. The focus in this paper is five general algorithm forms: random search, simultaneous perturbation stochastic approximation, simulated annealing, evolutionary strategies, and genetic algorithms. We summarize the available theoretical results on rates of convergence for the five algorithm forms and then use the theoretical results to draw some preliminary conclusions on the relative efficiency. Our aim is to sort out some of the competing claims of efficiency and to suggest a structure for comparison that is more general and transferable than the usual problem-specific numerical studies.

[1]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[2]  N. Metropolis,et al.  Equation of state calculations by fast computing machines , 1953 .

[3]  Ing Rj Ser.  Approximation Theorems of Mathematical Statistics , 1980 .

[4]  Marius Iosifescu,et al.  Finite Markov Processes and Their Applications , 1981 .

[5]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[6]  G. Rappl.  On Linear Convergence of a Class of Random Search Algorithms , 1989 .

[7]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[8]  S. Mitter,et al.  Metropolis-type annealing algorithms for global optimization in R d , 1993 .

[9]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: On the Benefits of Sex the (/, ) Theory , 1995, Evolutionary Computation.

[10]  Joe Suzuki,et al.  A Markov chain analysis on simple genetic algorithms , 1995, IEEE Trans. Syst. Man Cybern..

[11]  G. Unter Rudolph.  Convergence Rates of Evolutionary Algorithms for a Class of Convex Objective Functions , 1997 .

[12]  D. C. Chin,et al.  Comparative study of stochastic algorithms for system optimization based on gradient approximations , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[13]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[14]  J. Dippon,et al.  Weighted Means in Stochastic Approximation of Minima , 1997 .

[15]  XI FachbereichInformatik.  Finite Markov Chain Results in Evolutionary Computation: a Tour D'horizon , 1998 .

[16]  Gang George Yin.  Rates of Convergence for a Class of Global Stochastic Optimization Algorithms , 1999, SIAM J. Optim..

[17]  László Gerencsér,et al.  Convergence rate of moments in stochastic approximation with simultaneous perturbation gradient approximation and resetting , 1999, IEEE Trans. Autom. Control..

[18]  L. Gerencsér,et al.  SPSA in noise free optimization , 2000, Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No.00CH36334).

[19]  James C. Spall,et al.  Adaptive stochastic approximation by the simultaneous perturbation method , 2000, IEEE Trans. Autom. Control..

[20]  D. R. Stark,et al.  Computable bounds on the rate of convergence in evolutionary computation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[21]  Dirk V. Arnold,et al.  Noisy Optimization With Evolution Strategies , 2002, Genetic Algorithms and Evolutionary Computation.

[22]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.