Theoretical framework for comparing several popular stochastic optimization approaches

This paper establishes a framework for formal comparisons of several leading optimization algorithms, establishing guidance to practitioners for when to use or not to use a particular method. The focus in this paper are five general algorithm forms: random search, simultaneous perturbation stochastic approximation, simulated annealing, evolutionary strategies, and genetic algorithms. We summarize the available theoretical results on rates of convergence for the five algorithm forms and then use the theoretical results to draw some preliminary conclusions on the relative efficiency. Our aim is to sort out some of the competing claims of efficiency and to suggest a structure for comparison that is more general and transferable than the usual problem-specific numerical studies.

[1]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[2]  N. Metropolis,et al.  Equation of state calculations by fast computing machines , 1953 .

[3]  Marius Iosifescu,et al.  Finite Markov Processes and Their Applications , 2007 .

[4]  R. Serfling Approximation Theorems of Mathematical Statistics , 1980 .

[5]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[6]  G. Rappl On Linear Convergence of a Class of Random Search Algorithms , 1989 .

[7]  J. Spall Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[8]  S. Mitter,et al.  Metropolis-type annealing algorithms for global optimization in R d , 1993 .

[9]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: On the Benefits of Sex the (/, ) Theory , 1995, Evolutionary Computation.

[10]  Joe Suzuki A Markov chain analysis on simple genetic algorithms , 1995, IEEE Trans. Syst. Man Cybern..

[11]  Michael D. Vose Logarithmic Convergence of Random Heuristic Search , 1996, Evolutionary Computation.

[12]  Daniel C. Chin Comparative study of stochastic algorithms for system optimization based on gradient approximations , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[13]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[14]  Weighted Means in Stochastic Approximation of Minima , 1997 .

[15]  Günter Rudolph Finite Markov Chain Results in Evolutionary Computation: A Tour d'Horizon , 1998, Fundam. Informaticae.

[16]  Gian Luigi Ferrari,et al.  Parameterized Structured Operational Semantics , 1998, Fundam. Informaticae.

[17]  Gang George Yin Rates of Convergence for a Class of Global Stochastic Optimization Algorithms , 1999, SIAM J. Optim..

[18]  László Gerencsér,et al.  Convergence rate of moments in stochastic approximation with simultaneous perturbation gradient approximation and resetting , 1999, IEEE Trans. Autom. Control..

[19]  SPSA in noise free optimization , 2000, Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No.00CH36334).

[20]  James C. Spall,et al.  Adaptive stochastic approximation by the simultaneous perturbation method , 2000, IEEE Trans. Autom. Control..

[21]  D. R. Stark,et al.  Computable bounds on the rate of convergence in evolutionary computation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[22]  Global random optimization by simultaneous perturbation stochastic approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[23]  Dirk V. Arnold Noisy Optimization With Evolution Strategies , 2002, Genetic Algorithms and Evolutionary Computation.

[24]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization , 2003 .