Search based software testing of object-oriented containers

Automatic software testing tools are still far from ideal for real world object-oriented (OO) software. The use of nature inspired search algorithms for this problem has been investigated recently. Testing complex data structures (e.g., containers) is very challenging since testing software with simple states is already hard. Because containers are used in almost every type of software, their reliability is of utmost importance. Hence, this paper focuses on the difficulties of testing container classes with nature inspired search algorithms. We will first describe how input data can be automatically generated for testing Java containers. Input space reductions and a novel testability transformation are presented to aid the search algorithms. Different search algorithms are then considered and studied in order to understand when and why a search algorithm is effective for a testing problem. In our experiments, these nature inspired search algorithms seem to give better results than the traditional techniques described in literature. Besides, the problem of minimising the length of the test sequences is also addressed. Finally, some open research questions are given.

[1]  David Notkin,et al.  Rostra: a framework for detecting redundant object-oriented unit tests , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[2]  Daniel Hoffman,et al.  Programmatic testing of the Standard Template Library containers , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[3]  Mark Harman,et al.  Side-effect removal transformation , 2001, Proceedings 9th International Workshop on Program Comprehension. IWPC 2001.

[4]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[5]  Xin Yao,et al.  On Test Data Generation of Object-Oriented Software , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[6]  Xin Yao,et al.  Estimation of distribution algorithms for testing object oriented software , 2007, 2007 IEEE Congress on Evolutionary Computation.

[7]  Yoonsik Cheon,et al.  A specification-based fitness function for evolutionary testing of object-oriented programs , 2006, GECCO '06.

[8]  Jin-Cherng Lin,et al.  Automatic test data generation for path testing using GAs , 2001, Inf. Sci..

[9]  A. Dickson.  On Evolution , 1884, Science.

[10]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[11]  Phil McMinn,et al.  Evolutionary testing of state-based programs , 2005, GECCO '05.

[12]  Alessandro Orso,et al.  Automated Testing of Classes , 2000, ISSTA '00.

[13]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[14]  Xin Yao,et al.  Stochastic ranking for constrained evolutionary optimization , 2000, IEEE Trans. Evol. Comput..

[15]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.

[16]  Yoichi Hayashi,et al.  Neural expert system using fuzzy teaching input and its application to medical diagnosis , 1994 .

[17]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[18]  Joachim Wegener,et al.  Evolutionary Unit Testing Of Object-Oriented Software Using A Hybrid Evolutionary Algorithm , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[19]  Yoonsik Cheon,et al.  A Complete Automation of Unit Testing for Java Programs , 2005, Software Engineering Research and Practice.

[20]  Xin Yao,et al.  A Memetic Algorithm for test data generation of Object-Oriented software , 2007, 2007 IEEE Congress on Evolutionary Computation.

[21]  James Miller,et al.  Automatic test data generation using genetic algorithm and program dependence graphs , 2006, Inf. Softw. Technol..

[22]  Hélène Waeselynck,et al.  Simulated annealing applied to test generation: landscape characterization and stopping criteria , 2007, Empirical Software Engineering.

[23]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[24]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[25]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[26]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[27]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[28]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[29]  Sarfraz Khurshid,et al.  TestEra A Novel Framework for Testing Java Programs y , 2003 .

[30]  Mark Harman,et al.  Improving Evolutionary Testing By Flag Removal , 2002, GECCO.

[31]  B. G. Geetha,et al.  AUTOMATIC TEST DATA GENERATION USING GENETIC ALGORITHM AND PROGRAM DEPENDENCE GRAPH , 2010 .

[32]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[33]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[34]  Anthony Cox,et al.  Textual views of source code to support comprehension , 2005, 13th International Workshop on Program Comprehension (IWPC'05).

[35]  Pablo Moscato,et al.  On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts : Towards Memetic Algorithms , 1989 .

[36]  Tsong Yueh Chen,et al.  On the divide-and-conquer approach towards test suite reduction , 2003, Inf. Sci..

[37]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[38]  Philip McMinn,et al.  Evolutionary search for test data in the presence of state behaviour , 2005 .

[39]  Gregory Tassey,et al.  Prepared for what , 2007 .

[40]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[41]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[42]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[43]  Mark Harman,et al.  Testability Transformation for Efficient Automated Test Data Search in the Presence of Nesting , 2005 .

[44]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[45]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[46]  Stefan Wappler,et al.  Using evolutionary algorithms for the unit testing of object-oriented software , 2005, GECCO '05.

[47]  Daniel Hoffman,et al.  Data coverage testing of programs for container classes , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[48]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[49]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[50]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[51]  André Baresel,et al.  Evolutionary Testing of Flag Conditions , 2003, GECCO.

[52]  McMinnPhil.  Search-based software test data generation: a survey , 2004 .

[53]  Marcelo d'Amorim,et al.  An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[54]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[55]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[56]  Joachim Wegener,et al.  Evolutionary unit testing of object-oriented software using strongly-typed genetic programming , 2006, GECCO '06.

[57]  Michael Ellims,et al.  The Economics of Unit Testing , 2006, Empirical Software Engineering.

[58]  Mauricio G. C. Resende,et al.  Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[59]  Marek Reformat,et al.  Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques , 2007, Empirical Software Engineering.

[60]  John A. Clark,et al.  Formulating software engineering as a search problem , 2003, IEE Proc. Softw..

[61]  Xin Yao,et al.  Simulated annealing with extended neighbourhood , 1991, Int. J. Comput. Math..

[62]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[63]  Phil McMinn,et al.  Hybridizing Evolutionary Testing with the Chaining Approach , 2004, GECCO.

[64]  Phil McMinn,et al.  The State Problem for Evolutionary Testing , 2003, GECCO.

[65]  B. C. Brookes,et al.  Information Sciences , 2020, Cognitive Skills You Need for the 21st Century.