Benchmarking Optimization Algorithms: An Open Source Framework for the Traveling Salesman Problem

We introduce an experimentation procedure for evaluating and comparing optimization algorithms based on the Traveling Salesman Problem (TSP). We argue that end-of-run results alone do not give sufficient information about an algorithm's performance, so our approach analyzes the algorithm's progress over time. Comparisons of performance curves in diagrams can be formalized by comparing the areas under them. Algorithms can be ranked according to a performance metric. Rankings based on different metrics can then be aggregated into a global ranking, which provides a quick overview of the quality of algorithms in comparison. An open source software framework, the TSP Suite, applies this experimental procedure to the TSP. The framework can support researchers in implementing TSP solvers, unit testing them, and running experiments in a parallel and distributed fashion. It also has an evaluator component, which implements the proposed evaluation process and produces detailed reports. We test the approach by using the TSP Suite to benchmark several local search and evolutionary computation methods. This results in a large set of baseline data, which will be made available to the research community. Our experiments show that the tested pure global optimization algorithms are outperformed by local search, but the best results come from hybrid algorithms.

[1]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[2]  Colin R. Reeves,et al.  Evolutionary computation: a unified approach , 2007, Genetic Programming and Evolvable Machines.

[3]  Thomas Stützle,et al.  Ant colony optimization: artificial ants as a computational intelligence technique , 2006 .

[4]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[5]  Thomas Stützle,et al.  Evaluating Las Vegas Algorithms: Pitfalls and Remedies , 1998, UAI.

[6]  ATSPDavid S. JohnsonAT Experimental Analysis of Heuristics for the Stsp , 2001 .

[7]  Pierre Hansen,et al.  Variable neighbourhood search: methods and applications , 2010, Ann. Oper. Res..

[8]  Gerhard Reinelt,et al.  Traveling Salesman Problem , 2012 .

[9]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study , 2007 .

[10]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[11]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[12]  Xin Yao,et al.  Evolutionary Optimization , 2002 .

[13]  O. J. Dunn Multiple Comparisons among Means , 1961 .

[14]  Dirk Sudholt,et al.  Experimental supplements to the theoretical analysis of migration in the Island model , 2010, PPSN 2010.

[15]  Gerhard Reinelt,et al.  TSPLIB - A Traveling Salesman Problem Library , 1991, INFORMS J. Comput..

[16]  Thomas Weise,et al.  Global Optimization Algorithms -- Theory and Application , 2009 .

[17]  Mark S. Boddy,et al.  Solving Time-Dependent Planning Problems , 1989, IJCAI.

[18]  L. Darrell Whitley,et al.  Constant time steepest descent local search with lookahead for NK-landscapes and MAX-kSAT , 2012, GECCO '12.

[19]  Yew-Soon Ong,et al.  Memetic Computation—Past, Present & Future [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[20]  Shane Legg,et al.  Fitness uniform optimization , 2006, IEEE Transactions on Evolutionary Computation.

[21]  Gregory Gutin,et al.  The traveling salesman problem , 2006, Discret. Optim..

[22]  D. Fogel An evolutionary approach to the traveling salesman problem , 1988, Biological Cybernetics.

[23]  Eugene L. Lawler,et al.  The Traveling Salesman Problem: A Guided Tour of Combinatorial Optimization , 1985 .

[24]  Jeffrey Horn,et al.  Handbook of evolutionary computation , 1997 .

[25]  Shigeyoshi Tsutsui,et al.  Parallelization of an Evolutionary Algorithm on a Platform with Multi-core Processors , 2009, Artificial Evolution.

[26]  Michael Guntsch,et al.  Applying Population Based ACO to Dynamic Optimization Problems , 2002, Ant Algorithms.

[27]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[28]  J. Pach,et al.  Wiley‐Interscience Series in Discrete Mathematics and Optimization , 2011 .

[29]  David S. Johnson,et al.  The Traveling Salesman Problem: A Case Study in Local Optimization , 2008 .

[30]  Shigeyoshi Tsutsui Probabilistic Model-Building Genetic Algorithms in Permutation Representation Domain Using Edge Histogram , 2002, PPSN.

[31]  Xin Yao,et al.  Analysis of Computational Time of Simple Estimation of Distribution Algorithms , 2010, IEEE Transactions on Evolutionary Computation.

[33]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[34]  Brian W. Kernighan,et al.  An Effective Heuristic Algorithm for the Traveling-Salesman Problem , 1973, Oper. Res..

[35]  Raymond Chiong,et al.  SDGP: A developmental approach for traveling salesman problems , 2013, 2013 IEEE Symposium on Computational Intelligence in Production and Logistics Systems (CIPLS).

[36]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[37]  Jörg Lässig,et al.  Threshold-selecting strategy for best possible ground state detection with genetic algorithms. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[38]  Zbigniew Michalewicz,et al.  Evolutionary Optimization , 2012, Variants of Evolutionary Algorithms for Real-World Applications.

[39]  N. Hansen,et al.  Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup , 2010 .

[40]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[41]  David S. Johnson,et al.  Experimental Analysis of Heuristics for the STSP , 2007 .

[42]  Pedro Larrañaga,et al.  Genetic Algorithms for the Travelling Salesman Problem: A Review of Representations and Operators , 1999, Artificial Intelligence Review.

[43]  Thomas Bartz-Beielstein,et al.  SPOT: An R Package For Automatic and Interactive Tuning of Optimization Algorithms by Sequential Parameter Optimization , 2010, ArXiv.

[44]  L. Darrell Whitley,et al.  Scheduling Problems and Traveling Salesmen: The Genetic Edge Recombination Operator , 1989, International Conference on Genetic Algorithms.

[45]  Holger H. Hoos,et al.  UBCSAT: An Implementation and Experimentation Environment for SLS Algorithms for SAT & MAX-SAT , 2004, SAT.

[46]  Keld Helsgaun,et al.  General k-opt submoves for the Lin–Kernighan TSP heuristic , 2009, Math. Program. Comput..

[47]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study (Princeton Series in Applied Mathematics) , 2007 .

[48]  Xin Yao,et al.  Frequency Fitness Assignment , 2014, IEEE Transactions on Evolutionary Computation.