Designing and reporting on computational experiments with heuristic methods

This article discusses the design of computational experiments to test heuristic methods and provides reporting guidelines for such experimentation. The goal is to promote thoughtful, well-planned, and extensive testing of heuristics, full disclosure of experimental conditions, and integrity in and reproducibility of the reported results.

[1]  Richard E. Nance,et al.  A statistical technique for comparing heuristics: an example from capacity assignment strategies in computer network design , 1987, CACM.

[2]  Ronald L. Rardin,et al.  Development of a Parametric Generating Procedure for Integer Programming Test Problems , 1977, JACM.

[3]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[4]  Richard S. Barr,et al.  Network Reoptimization Algorithms: A Statistically Designed Comparison , 1993, INFORMS J. Comput..

[5]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[6]  Richard M. Karp,et al.  The Traveling-Salesman Problem and Minimum Spanning Trees , 1970, Oper. Res..

[7]  G. Cornuéjols,et al.  A comparison of heuristics and relaxations for the capacitated plant location problem , 1991 .

[8]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[9]  David B. Shmoys,et al.  A Best Possible Heuristic for the k-Center Problem , 1985, Math. Oper. Res..

[10]  P. B. Saunders,et al.  Methodology and analysis for comparing discrete linear l1 approximation codes , 1977 .

[11]  David S. Johnson,et al.  Local Optimization and the Traveling Salesman Problem , 1990, ICALP.

[12]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[13]  Stephen G. Nash,et al.  Guidelines for reporting results of computational experiments. Report of the ad hoc committee , 1991, Math. Program..

[14]  Catherine C. McGeoch.  Feature Article - Toward an Experimental Method for Algorithm Simulation , 1996, INFORMS J. Comput..

[15]  A. Frieze,et al.  A simple heuristic for the p-centre problem , 1985 .

[16]  Gilbert Laporte,et al.  A Tabu Search Heuristic for the Vehicle Routing Problem , 1991 .

[17]  Catherine C. McGeoch.  Analyzing algorithms by simulation: variance reduction techniques and simulation speedups , 1992, CSUR.

[18]  N. Biggs.  THE TRAVELING SALESMAN PROBLEM A Guided Tour of Combinatorial Optimization , 1986 .

[19]  John M. Mulvey,et al.  Evaluating Mathematical Programming Techniques , 1982 .

[20]  Colin R. Reeves.  Evaluation of heuristic performance , 1993 .

[21]  Brian W. Kernighan,et al.  An Effective Heuristic Algorithm for the Traveling-Salesman Problem , 1973, Oper. Res..

[22]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[23]  John M. Mulvey,et al.  Guidelines for Reporting Computational Experiments in Mathematical Programming , 1978 .

[24]  Panos M. Pardalos,et al.  A Collection of Test Problems for Constrained Global Optimization Algorithms , 1990, Lecture Notes in Computer Science.

[25]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[26]  George E. P. Box,et al.  Evolutionary Operation: A Statistical Method for Process Improvement , 1969 .

[27]  Benjamin W. Lin,et al.  Controlled Experimental Design for Statistical Comparison of Integer Programming Algorithms , 1979 .

[28]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[29]  Jeffrey L. Arthur,et al.  An algorithm for generating minimum cost network flow problems with specific structure and known optimal solutions , 1994, Networks.

[30]  James P. Kelly,et al.  Large-scale controlled rounding using tabu search with strategic oscillation , 1993, Ann. Oper. Res..

[31]  J. M. Mulvey,et al.  A Critical Review of Comparisons of Mathematical Programming Algorithms and Software (1953-1977). , 1978, Journal of research of the National Bureau of Standards.

[32]  Mauricio G. C. Resende,et al.  A Greedy Randomized Adaptive Search Procedure for Maximum Independent Set , 1994, Oper. Res..

[33]  J. S. Ivey,et al.  Nelder-Mead simplex modifications for simulation optimization , 1996 .

[34]  Richard S. Barr,et al.  Parallel Simplex for Large Pure Network Problems: Computational Testing and Sources of Speedup , 1994, Oper. Res..

[35]  Richard P. O’Neill.  A Comparison of Real-World Linear Programs and Their Randomly Generated Analogs , 1982 .

[36]  Celso C. Ribeiro,et al.  A GRASP for graph planarization , 1997, Networks.

[37]  R. Battiti,et al.  Simulated annealing and Tabu search in the long run: A comparison on QAP tasks☆ , 1994 .

[38]  Harvey J. Greenberg.  Computational Testing: Why, How and How Much , 1990, INFORMS J. Comput..

[39]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[40]  Catherine C. McGeoch.  Experimental analysis of algorithms , 1986 .

[41]  Mauricio G. C. Resende,et al.  Greedy Randomized Adaptive Search Procedures , 1995, J. Glob. Optim..

[42]  Gerhard Reinelt,et al.  TSPLIB - A Traveling Salesman Problem Library , 1991, INFORMS J. Comput..

[43]  James R. Evans,et al.  Heuristic methods and applications: A categorized survey , 1989 .

[44]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[45]  Jan Karel Lenstra,et al.  A Computational Study of Local Search Algorithms for Job Shop Scheduling , 1994, INFORMS J. Comput..

[46]  William R. Stewart.  Accelerated branch exchange heuristics for symmetric traveling salesman problems , 1987, Networks.

[47]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[48]  C. Reeves.  Modern heuristic techniques for combinatorial problems , 1993 .

[49]  Fred Glover,et al.  A Computation Study on Start Procedures, Basis Change Criteria, and Solution Algorithms for Transportation Problems , 1974 .

[50]  M. Fisher.  Worst-Case Analysis of Heuristic Algorithms , 1980 .

[51]  Richard M. Karp,et al.  The traveling-salesman problem and minimum spanning trees: Part II , 1971, Math. Program..

[52]  Paul Bratley,et al.  A guide to simulation , 1983 .

[53]  Anne H. Soukhanov,et al.  The american heritage dictionary of the english language , 1992 .

[54]  Jeffrey L. Arthur,et al.  Generating Travelling-Salesman Problems with Known Optimal Tours , 1988 .

[55]  Paul Juell,et al.  Neural Networks for Selective Vehicle Routing Heuristics , 1990, INFORMS J. Comput..

[56]  P. John.  Statistical Design and Analysis of Experiments , 1971 .

[57]  Joseph Cheriyan,et al.  An Empirical Study of Min Cost Flow Algorithms for the Minimum-Cost Flow Problem , 1993, Network Flows And Matching.

[58]  L. A. Oley.  Book reviewMathematical programming techniques: Affiliated East-West Press PVT, 104 Nirmal Tower, 26 Barakhamba Road, New Delhi, 1984, 719 pages, RS57.00 (India) , 1985 .

[59]  Bruce L. Golden,et al.  Experimentation in optimization , 1986 .

[60]  Kenneth Steiglitz,et al.  Optimal Design of Offshore Natural-Gas Pipeline Systems , 1970, Oper. Res..

[61]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[62]  Darwin Klingman,et al.  NETGEN: A Program for Generating Large Scale Capacitated Assignment, Transportation, and Minimum Cost Flow Network Problems , 1974 .

[63]  Ronald L. Rardin,et al.  Test Problems for Computational Experiments -- Issues and Techniques , 1982 .

[64]  Ravindra K. Ahuja,et al.  Network Flows: Theory, Algorithms, and Applications , 1993 .

[65]  John Knox,et al.  Tabu search performance on the symmetric traveling salesman problem , 1994, Comput. Oper. Res..

[66]  James P. Kelly,et al.  Cell suppression: Disclosure protection for sensitive tabular data , 1992, Networks.

[67]  Richard S. Barr,et al.  Feature Article - Reporting Computational Experiments with Parallel Algorithms: Issues, Measures, and Experts' Opinions , 1993, INFORMS J. Comput..