Performance analysis of dynamic optimization algorithms using relative error distance

Abstract Quantification of the performance of algorithms that solve dynamic optimization problems (DOPs) is challenging, since the fitness landscape changes over time. Popular performance measures for DOPs do not adequately account for ongoing fitness landscape scale changes, and often yield a confounded view of performance. Similarly, most popular measures do not allow for fair performance comparisons across multiple instances of the same problem type nor across different types of problems, since performance values are not normalized. Many measures also assume normally distributed input data values, while in reality the necessary conditions for data normality are often not satisfied. The majority of measures also fail to capture the notion of performance variance over time. This paper proposes a new performance measure for DOPs, namely the relative error distance. The measure shows how close to optimal an algorithm performs by considering the multi-dimensional distance between the vector comprising the normalized performance scores for specific algorithm iterations of interest, and the theoretical point of best possible performance. The new measure does not assume normally distributed performance data across fitness landscape changes, is resilient against fitness landscape scale changes, better incorporates performance variance across fitness landscape changes into a single scalar value, and allows easier algorithm comparisons using established nonparametric statistical methods.

[1]  Ponnuthurai N. Suganthan,et al.  An Adaptive Differential Evolution Algorithm With Novel Mutation and Crossover Strategies for Global Numerical Optimization , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Tim M. Blackwell,et al.  Swarms in Dynamic Environments , 2003, GECCO.

[3]  Jürgen Branke,et al.  Memory enhanced evolutionary algorithms for changing optimization problems , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[4]  Andries Petrus Engelbrecht,et al.  The Effect of Probability Distributions on the Performance of Quantum Particle Swarm Optimization for Solving Dynamic Optimization Problems , 2015, 2015 IEEE Symposium Series on Computational Intelligence.

[5]  Peter J. Bentley,et al.  Don't push me! Collision-avoiding swarms , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[6]  Elizabeth F. Wanner,et al.  Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances , 2019, Journal of Heuristics.

[7]  Andries Petrus Engelbrecht,et al.  Analysis of global information sharing in hyper-heuristics for different dynamic environments , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[8]  Jürgen Branke,et al.  Evolutionary Optimization in Dynamic Environments , 2001, Genetic Algorithms and Evolutionary Computation.

[9]  Andries Petrus Engelbrecht,et al.  Optimal parameter regions and the time-dependence of control parameter values for the particle swarm optimization algorithm , 2018, Swarm Evol. Comput..

[10]  Edmund K. Burke,et al.  A greedy hyper-heuristic in dynamic environments , 2009, GECCO '09.

[11]  Russell C. Eberhart,et al.  Tracking and optimizing dynamic systems with particle swarms , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[12]  Zbigniew Michalewicz,et al.  Searching for optima in non-stationary environments , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[13]  Charu C. Aggarwal,et al.  On the Surprising Behavior of Distance Metrics in High Dimensional Spaces , 2001, ICDT.

[14]  Andries Petrus Engelbrecht,et al.  Alternative hyper-heuristic strategies for multi-method global optimization , 2010, IEEE Congress on Evolutionary Computation.

[15]  Andries Petrus Engelbrecht,et al.  Using Competitive Population Evaluation in a differential evolution algorithm for dynamic environments , 2012, Eur. J. Oper. Res..

[16]  Carlos A. Coello Coello,et al.  On the adaptation of the mutation scale factor in differential evolution , 2015, Optim. Lett..

[17]  P. N. Suganthan,et al.  Differential Evolution: A Survey of the State-of-the-Art , 2011, IEEE Transactions on Evolutionary Computation.

[18]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[19]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (3rd ed.) , 1996 .

[20]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[21]  Andries Petrus Engelbrecht,et al.  Fundamentals of Computational Swarm Intelligence , 2005 .

[22]  Jürgen Branke,et al.  Multi-swarm Optimization in Dynamic Environments , 2004, EvoWorkshops.

[23]  R. Luce,et al.  Individual Choice Behavior: A Theoretical Analysis. , 1960 .

[24]  Andries Petrus Engelbrecht,et al.  Multi-method algorithms: Investigating the entity-to-algorithm allocation problem , 2013, 2013 IEEE Congress on Evolutionary Computation.

[25]  Jürgen Branke,et al.  Multiswarms, exclusion, and anti-convergence in dynamic environments , 2006, IEEE Transactions on Evolutionary Computation.

[26]  M. Friedman The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[27]  A. Sima Etaner-Uyar,et al.  An Investigation of Selection Hyper-heuristics in Dynamic Environments , 2011, EvoApplications.

[28]  Karsten Weicker,et al.  Performance Measures for Dynamic Environments , 2002, PPSN.

[29]  Changhe Li,et al.  A survey of swarm intelligence for dynamic optimization: Algorithms and applications , 2017, Swarm Evol. Comput..

[30]  M. Chowdhury,et al.  Benchmarks for testing evolutionary algorithms , 1998 .

[31]  T. Back,et al.  On the behavior of evolutionary algorithms in dynamic environments , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[32]  A. Sima Etaner-Uyar,et al.  An Ant-Based Selection Hyper-heuristic for Dynamic Environments , 2013, EvoApplications.

[33]  Peter J. Bentley,et al.  Dynamic Search With Charged Swarms , 2002, GECCO.

[34]  Xin Yao,et al.  Benchmark Generator for CEC'2009 Competition on Dynamic Optimization , 2008 .

[35]  Andries Petrus Engelbrecht,et al.  Evolutionary and swarm-intelligence algorithms through monadic composition , 2019, GECCO.

[36]  Carlos Cruz,et al.  Optimization in dynamic environments: a survey on problems, methods and measures , 2011, Soft Comput..

[37]  Carlos A. Coello Coello,et al.  A comparative study of differential evolution variants for global optimization , 2006, GECCO.

[38]  Andries Petrus Engelbrecht,et al.  On the optimality of particle swarm parameters in dynamic environments , 2013, 2013 IEEE Congress on Evolutionary Computation.

[39]  Andries Petrus Engelbrecht,et al.  Towards a more complete classification system for dynamically changing environments , 2012, 2012 IEEE Congress on Evolutionary Computation.

[40]  C. Spearman ‘FOOTRULE’ FOR MEASURING CORRELATION , 1906 .

[41]  Hartmut Schmeck,et al.  Designing evolutionary algorithms for dynamic optimization problems , 2003 .

[42]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[43]  Peter J. Angeline,et al.  Tracking Extrema in Dynamic Environments , 1997, Evolutionary Programming.

[44]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[45]  John J. Grefenstette,et al.  Genetic Algorithms for Changing Environments , 1992, PPSN.

[46]  Jonathan Goldstein,et al.  When Is ''Nearest Neighbor'' Meaningful? , 1999, ICDT.

[47]  Andries Petrus Engelbrecht,et al.  Analysis of hyper-heuristic performance in different dynamic environments , 2014, 2014 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE).

[48]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[49]  R. Eberhart,et al.  Comparing inertia weights and constriction factors in particle swarm optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[50]  Ponnuthurai N. Suganthan,et al.  Recent advances in differential evolution - An updated survey , 2016, Swarm Evol. Comput..

[51]  M. Friedman A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[52]  Andries Petrus Engelbrecht,et al.  Particle swarm stability: a theoretical extension using the non-stagnate distribution assumption , 2018, Swarm Intelligence.

[53]  Haluk Topcuoglu,et al.  A hyper-heuristic based framework for dynamic optimization problems , 2014, Appl. Soft Comput..

[54]  J. Shaffer Modified Sequentially Rejective Multiple Test Procedures , 1986 .

[55]  Francisco Herrera,et al.  A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[56]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[57]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[58]  Andries Petrus Engelbrecht,et al.  Analysis of selection hyper-heuristics for population-based meta-heuristics in real-valued dynamic optimization , 2018, Swarm Evol. Comput..

[59]  John J. Grefenstette,et al.  Genetic Algorithms for Tracking Changing Environments , 1993, ICGA.

[60]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[61]  Shengxiang Yang,et al.  Evolutionary dynamic optimization: A survey of the state of the art , 2012, Swarm Evol. Comput..

[62]  Andries Petrus Engelbrecht,et al.  Heuristic space diversity control for improved meta-hyper-heuristic performance , 2015, Inf. Sci..