DG2: A Faster and More Accurate Differential Grouping for Large-Scale Black-Box Optimization

Identification of variable interaction is essential for an efficient implementation of a divide-and-conquer algorithm for large-scale black-box optimization. In this paper, we propose an improved variant of the differential grouping (DG) algorithm, which has a better efficiency and grouping accuracy. The proposed algorithm, DG2, finds a reliable threshold value by estimating the magnitude of roundoff errors. With respect to efficiency, DG2 reuses the sample points that are generated for detecting interactions and saves up to half of the computational resources on fully separable functions. We mathematically show that the new sampling technique achieves the lower bound with respect to the number of function evaluations. Unlike its predecessor, DG2 checks all possible pairs of variables for interactions and has the capacity to identify overlapping components of an objective function. On the accuracy aspect, DG2 outperforms the state-of-the-art decomposition methods on the latest large-scale continuous optimization benchmark suites. DG2 also performs reliably in the presence of imbalance among contribution of components in an objective function. Another major advantage of DG2 is the automatic calculation of its threshold parameter ( $\epsilon $ ), which makes it parameter-free. Finally, the experimental results show that when DG2 is used within a cooperative co-evolutionary framework, it can generate competitive results as compared to several state-of-the-art algorithms.

[1]  Antonio LaTorre,et al.  Large scale global optimization: Experimental results with MOS-based hybrid algorithms , 2013, 2013 IEEE Congress on Evolutionary Computation.

[2]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[3]  X. Yao,et al.  Scaling up fast evolutionary programming with cooperative coevolution , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[4]  Arthur C. Sanderson,et al.  JADE: Adaptive Differential Evolution With Optional External Archive , 2009, IEEE Transactions on Evolutionary Computation.

[5]  Jing J. Liang,et al.  Dynamic multi-swarm particle swarm optimizer with local search for Large Scale Global Optimization , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[6]  Yanchun Liang,et al.  A cooperative particle swarm optimizer with statistical variable interdependence learning , 2012, Inf. Sci..

[7]  Bin Li,et al.  Variance priority based cooperative co-evolution differential evolution for large scale global optimization , 2009, 2009 IEEE Congress on Evolutionary Computation.

[8]  Zhenyu Yang,et al.  Large-Scale Global Optimization Using Cooperative Coevolution with Variable Interaction Learning , 2010, PPSN.

[9]  Robert M. Corless,et al.  A Graduate Introduction to Numerical Methods , 2013 .

[10]  Ke Tang,et al.  Scaling Up Covariance Matrix Adaptation Evolution Strategy Using Cooperative Coevolution , 2013, IDEAL.

[11]  Xiaodong Li,et al.  Cooperative Co-Evolution With Differential Grouping for Large Scale Optimization , 2014, IEEE Transactions on Evolutionary Computation.

[12]  Wei Zeng,et al.  A Dual-System Variable-Grain Cooperative Coevolutionary Algorithm: Satellite-Module Layout Design , 2010, IEEE Transactions on Evolutionary Computation.

[13]  Yuan Sun,et al.  Extended Differential Grouping for Large Scale Global Optimization with Direct and Indirect Variable Interactions , 2015, GECCO.

[14]  W. Hager,et al.  Large Scale Optimization : State of the Art , 1993 .

[15]  Ponnuthurai Nagaratnam Suganthan,et al.  Benchmark Functions for the CEC'2013 Special Session and Competition on Large-Scale Global Optimization , 2008 .

[16]  Hande Y. Benson,et al.  A Comparative Study of Large-Scale Nonlinear Optimization Algorithms , 2003 .

[17]  Chun Chen,et al.  Multiple trajectory search for Large Scale Global Optimization , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[18]  Xiaodong Li,et al.  A novel hybridization of opposition-based learning and cooperative co-evolutionary for large-scale optimization , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[19]  Janez Brest,et al.  High-dimensional real-parameter optimization using Self-Adaptive Differential Evolution algorithm with population size reduction , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[20]  Pat H. Sterbenz,et al.  Floating-point computation , 1973 .

[21]  Janez Brest,et al.  Large Scale Global Optimization using Differential Evolution with self-adaptation and cooperative co-evolution , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[22]  Saman K. Halgamuge,et al.  On the Selection of Decomposition Methods for Large Scale Fully Non-separable Problems , 2015, GECCO.

[23]  Xin Yao,et al.  Self-adaptive differential evolution with neighborhood search , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[24]  Attila Szolnoki,et al.  Coevolutionary Games - A Mini Review , 2009, Biosyst..

[25]  Nikolaus Hansen,et al.  Evaluating the CMA Evolution Strategy on Multimodal Test Functions , 2004, PPSN.

[26]  Xin Yao,et al.  Large scale evolutionary optimization using cooperative coevolution , 2008, Inf. Sci..

[27]  Kalyanmoy Deb,et al.  Breaking the Billion-Variable Barrier in Real-World Optimization Using a Customized Evolutionary Algorithm , 2016, GECCO.

[28]  Christine A. Shoemaker,et al.  Local function approximation in evolutionary algorithms for the optimization of costly functions , 2004, IEEE Transactions on Evolutionary Computation.

[29]  KonagayaAkihiko,et al.  Inference of S-system models of genetic networks using a cooperative coevolutionary algorithm , 2005 .

[30]  Xiaodong Li,et al.  Effective decomposition of large-scale separable continuous functions for cooperative co-evolutionary algorithms , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[31]  Graham J. Williams,et al.  Big Data Opportunities and Challenges: Discussions from Data Analytics Perspectives [Discussion Forum] , 2014, IEEE Computational Intelligence Magazine.

[32]  Quoc V. Le,et al.  On optimization methods for deep learning , 2011, ICML.

[33]  Soontorn Oraintara,et al.  Dimensionality Reduction Based Optimization Algorithm for Sparse 3-D Image Reconstruction in Diffuse Optical Tomography , 2016, Scientific Reports.

[34]  Peter Tiño,et al.  Scaling Up Estimation of Distribution Algorithms for Continuous Optimization , 2011, IEEE Transactions on Evolutionary Computation.

[35]  Kenneth A. De Jong,et al.  A Cooperative Coevolutionary Approach to Function Optimization , 1994, PPSN.

[36]  Ata Kabán,et al.  Toward Large-Scale Continuous EDA: A Random Matrix Theory Perspective , 2013, Evolutionary Computation.

[37]  Antonio LaTorre,et al.  A comprehensive comparison of large scale global optimizers , 2015, Inf. Sci..

[38]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[39]  Shuhei Kimura,et al.  Inference of S-system models of genetic networks using a cooperative coevolutionary algorithm , 2005, Bioinform..

[40]  Andries Petrus Engelbrecht,et al.  A Cooperative approach to particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[41]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[42]  Junchi Yan,et al.  Two-stage based ensemble optimization framework for large-scale global optimization , 2013, Eur. J. Oper. Res..

[43]  Xiaodong Li,et al.  Smart use of computational resources based on contribution for cooperative co-evolutionary algorithms , 2011, GECCO '11.

[44]  Bin Li,et al.  Cooperative Coevolution with global search for large scale global optimization , 2012, 2012 IEEE Congress on Evolutionary Computation.

[45]  Xin Yao,et al.  Target shape design optimization by evolving B-splines with cooperative coevolution , 2016, Appl. Soft Comput..

[46]  Francisco Herrera,et al.  Editorial scalability of evolutionary algorithms and other metaheuristics for large-scale continuous optimization problems , 2011, Soft Comput..

[47]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[48]  Shahryar Rahnamayan,et al.  Cooperative Co-evolution with a new decomposition method for large-scale optimization , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[49]  Xiaodong Li,et al.  Benchmark Functions for the CEC'2010 Special Session and Competition on Large-Scale , 2009 .

[50]  Carlos A. Coello Coello,et al.  Use of cooperative coevolution for solving large scale multiobjective optimization problems , 2013, 2013 IEEE Congress on Evolutionary Computation.

[51]  Hongfei Teng,et al.  Cooperative Co-evolutionary Differential Evolution for Function Optimization , 2005, ICNC.

[52]  Xiaodong Li,et al.  Cooperative Co-evolution with delta grouping for large scale non-separable function optimization , 2010, IEEE Congress on Evolutionary Computation.

[53]  Xiaodong Li,et al.  A Competitive Divide-and-Conquer Algorithm for Unconstrained Large-Scale Black-Box Optimization , 2016, ACM Trans. Math. Softw..

[54]  Xin Yao,et al.  Multilevel cooperative coevolution for large scale optimization , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[55]  Xiaodong Li,et al.  A sensitivity analysis of contribution-based cooperative co-evolutionary algorithms , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[56]  Xiaodong Li,et al.  CBCC3 — A contribution-based cooperative co-evolutionary algorithm with improved exploration/exploitation balance , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[57]  Francisco Herrera,et al.  MA-SW-Chains: Memetic algorithm based on local search chains for large scale continuous global optimization , 2010, IEEE Congress on Evolutionary Computation.

[58]  Xiaodong Li,et al.  Designing benchmark problems for large-scale continuous optimization , 2015, Inf. Sci..

[59]  Yuping Wang,et al.  A novel cooperative coevolution for large scale global optimization , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[60]  Chao Wang,et al.  High-Dimensional Waveform Inversion With Cooperative Coevolutionary Differential Evolution Algorithm , 2012, IEEE Geoscience and Remote Sensing Letters.

[61]  Ruhul A. Sarker,et al.  Dependency Identification technique for large scale optimization problems , 2012, 2012 IEEE Congress on Evolutionary Computation.

[62]  Xiaodong Li,et al.  Cooperatively Coevolving Particle Swarms for Large Scale Optimization , 2012, IEEE Transactions on Evolutionary Computation.

[63]  Tapabrata Ray,et al.  A cooperative coevolutionary algorithm with Correlation based Adaptive Variable Partitioning , 2009, 2009 IEEE Congress on Evolutionary Computation.

[64]  Ilya Loshchilov,et al.  LM-CMA: An Alternative to L-BFGS for Large-Scale Black Box Optimization , 2015, Evolutionary Computation.

[65]  J. C. Townsend,et al.  Very Large Scale Optimization , 2000 .

[66]  Yaochu Jin,et al.  A Competitive Swarm Optimizer for Large Scale Optimization , 2015, IEEE Transactions on Cybernetics.

[67]  K. Deb,et al.  Optimal Scheduling of Casting Sequence Using Genetic Algorithms , 2003 .

[68]  Jun Wang,et al.  Cooperative Coevolution for Large-Scale Optimization Based on Kernel Fuzzy Clustering and Variable Trust Region Methods , 2014, IEEE Transactions on Fuzzy Systems.

[69]  Shahryar Rahnamayan,et al.  Metaheuristics in large-scale global continues optimization: A survey , 2015, Inf. Sci..

[70]  Xiaodong Li,et al.  Editorial for the special issue of Information Sciences Journal (ISJ) on "Nature-inspired algorithms for large scale global optimization" , 2015, Inf. Sci..