Local function approximation in evolutionary algorithms for the optimization of costly functions

We develop an approach for the optimization of continuous costly functions that uses a space-filling experimental design and local function approximation to reduce the number of function evaluations in an evolutionary algorithm. Our approach is to estimate the objective function value of an offspring by fitting a function approximation model over the k nearest previously evaluated points, where k=(d+1)(d+2)/2 and d is the dimension of the problem. The estimated function values are used to screen offspring to identify the most promising ones for function evaluation. To fit function approximation models, a symmetric Latin hypercube design (SLHD) is used to determine initial points for function evaluation. We compared the performance of an evolution strategy (ES) with local quadratic approximation, an ES with local cubic radial basis function (RBF) interpolation, an ES whose initial parent population comes from an SLHD, and a conventional ES. These algorithms were applied to a twelve-dimensional (12-D) groundwater bioremediation problem involving a complex nonlinear finite-element simulation model. The performances of these algorithms were also compared on the Dixon-Szego test functions and on the ten-dimensional (10-D) Rastrigin and Ackley test functions. All comparisons involve analysis of variance (ANOVA) and the computation of simultaneous confidence intervals. The results indicate that ES algorithms with local approximation were significantly better than conventional ES algorithms and ES algorithms initialized by SLHDs on all Dixon-Szego test functions except for Goldstein-Price. However, for the more difficult 10-D and 12-D functions, only the cubic RBF approach was successful in improving the performance of an ES. Moreover, the results also suggest that the cubic RBF approach is superior to the quadratic approximation approach on all test functions and the difference in performance is statistically significant for all test functions with dimension d/spl ges/4.

[1]  Lawrence J. Fogel,et al.  Toward Inductive Inference Automata , 1962, IFIP Congress.

[2]  John H. Holland,et al.  Outline for a Logical Theory of Adaptive Systems , 1962, JACM.

[3]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[4]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[5]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[6]  Dipl. Ing. Karl Heinz Kellermayer.  NUMERISCHE OPTIMIERUNG VON COMPUTER-MODELLEN MITTELS DER EVOLUTIONSSTRATEGIE Hans-Paul Schwefel Birkhäuser, Basel and Stuttgart, 1977 370 pages Hardback SF/48 ISBN 3-7643-0876-1 , 1977 .

[7]  D. Ackley.  A connectionist machine for genetic hillclimbing , 1987 .

[8]  Derek J. Pike,et al.  Empirical Model‐building and Response Surfaces. , 1988 .

[9]  C. Shoemaker,et al.  Dynamic optimal control for groundwater remediation with flexible management periods , 1992 .

[10]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[11]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[12]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[13]  R. H. Myers,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[14]  Thomas Bäck,et al.  Evolutionary computation: an overview , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[15]  S. Batill,et al.  Concurrent Subspace Optimization using gradient-enhanced neural network approximations , 1996 .

[16]  John E. Renaud,et al.  Response surface based, concurrent subspace optimization for multidisciplinary system design , 1996 .

[17]  M. Matsunami,et al.  An optimization method based on radial basis function , 1997 .

[18]  S. Varadarajan,et al.  Integration of design of experiments and artificial neural networks for achieving affordable concurrent design , 1997 .

[19]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[20]  D. Fogel.  Evolutionary algorithms in theory and practice , 1997, Complex..

[21]  Timothy M. Mauery,et al.  COMPARISON OF RESPONSE SURFACE AND KRIGING MODELS FOR MULTIDISCIPLINARY DESIGN OPTIMIZATION , 1998 .

[22]  Alain Ratle,et al.  Accelerating the Convergence of Evolutionary Algorithms by Fitness Landscape Approximation , 1998, PPSN.

[23]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[24]  Andy J. Keane,et al.  Metamodeling Techniques For Evolutionary Optimization of Computationally Expensive Problems: Promises and Limitations , 1999, GECCO.

[25]  A. Ratle.  Optimal sampling strategies for learning a fitness model , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[26]  M. Powell.  Recent research at Cambridge on radial basis functions , 1999 .

[27]  M. Matsunami,et al.  A combined method for the global optimization using radial basis function and deterministic approach , 1999 .

[28]  Christine A. Shoemaker,et al.  Comparison of Optimization Methods for Ground-Water Bioremediation , 1999 .

[29]  Kenny Q. Ye,et al.  Algorithmic construction of optimal symmetric Latin hypercube designs , 2000 .

[30]  Bernhard Sendhoff,et al.  On Evolutionary Optimization with Approximate Fitness Functions , 2000, GECCO.

[31]  K. Rasheed,et al.  An incremental-approximate-clustering approach for developing dynamic reduced models for design optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[32]  S. Batill,et al.  AN ITERATIVE CONCURRENT SUBSPACE ROBUST DESIGN FRAMEWORK , 2000 .

[33]  Weiyu Liu,et al.  GRADIENT-ENHANCED NEURAL NETWORK RESPONSE SURFACE APPROXIMATIONS , 2000 .

[34]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[35]  Mattias Björkman,et al.  Global Optimization of Costly Nonconvex Functions Using Radial Basis Functions , 2000 .

[36]  Yaochu Jin,et al.  Managing approximate models in evolutionary aerodynamic design optimization , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[37]  Hans-Martin Gutmann,et al.  On the Semi-norm of Radial Basis Function Interpolants , 2001, J. Approx. Theory.

[38]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[39]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[40]  Andy J. Keane,et al.  Evolutionary optimization for computationally expensive problems using Gaussian processes , 2001 .

[41]  Khaled Rasheed,et al.  Comparison Of Methods For Using Reduced Models To Speed Up Design Optimization , 2002, GECCO.

[42]  Bernhard Sendhoff,et al.  Fitness Approximation In Evolutionary Computation - a Survey , 2002, GECCO.

[43]  Bernhard Sendhoff,et al.  A framework for evolutionary optimization with approximate fitness functions , 2002, IEEE Trans. Evol. Comput..

[44]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[45]  Khaled Rasheed,et al.  Comparison of methods for developing dynamic reduced models for design optimization , 2002, Soft Comput..

[46]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..