Subtle higher order mutants

Context: Research has shown the majority of real faults are complex and cannot be simulated with traditional First Order Mutants (FOMs). Higher Order Mutants (HOMs), which are created by making multiple syntactic changes to the source code (i.e, combining multiple FOMs), can potentially be used to simulate realistic faults.Objective: Our goal is to find subtle HOMs, which we define to be HOMs that are not killed by a reference test suite that kills all the FOMs. Subtle HOMs represent cases where single faults interact by masking each other to produce complex faulty behavior. The fault-detection effectiveness of a test suite can be improved by adding test cases that detect subtle HOMs.Method: We developed six search-based techniques to find subtle HOMs: Genetic Algorithm, Local Search, Data-Interaction Guided Local Search, Test-Case Guided Local Search, Restricted Random Search and Restricted Enumeration. We then performed empirical studies to evaluate the ability of the search techniques to find subtle HOMs and investigated factors that impact the characteristics of the subtle HOMs.Results: Local Search and both the Guided Local Search techniques were more effective than the other techniques at finding subtle HOMs. The majority of subtle HOMs that were found resulted from combining mutated Java primitive operators.Conclusions: The proposed techniques are able to find subtle HOMs. Composing subtle HOMs of lower degrees is an effective way for finding new subtle HOMs of higher degrees.

[1]  William R. Cook,et al.  Safe composition of product lines , 2007, GPCE '07.

[2]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[3]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[4]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[5]  Lech Madeyski,et al.  Judy - a mutation testing tool for Java , 2010, IET Softw..

[6]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[7]  Fadi Wedyan,et al.  On generating mutants for AspectJ programs , 2012, Inf. Softw. Technol..

[8]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[9]  Mark Harman,et al.  Angels and monsters: an empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation , 2014, ASE.

[10]  L. Darrell Whitley,et al.  Comparing search techniques for finding subtle higher order mutants , 2014, GECCO.

[11]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[12]  Massimiliano Di Penta,et al.  Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments , 2011, IEEE Transactions on Software Engineering.

[13]  Andrea Arcuri,et al.  On the automation of fixing software bugs , 2008, ICSE Companion '08.

[14]  Tai-hoon Kim,et al.  Application of Genetic Algorithm in Software Testing , 2009 .

[15]  Sudipto Ghosh,et al.  An Exploratory Study of Higher Order Mutation Testing in Aspect-Oriented Programming , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[16]  A. Jefferson Offutt,et al.  The class-level mutants of MuJava , 2006, AST '06.

[17]  W. Eric Wong,et al.  Insights on Fault Interference for Programs with Multiple Bugs , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[18]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[19]  A. Jefferson Offutt,et al.  An experimental mutation system for Java , 2004, SOEN.

[20]  L. Darrell Whitley,et al.  Constructing subtle higher order mutants for Java and AspectJ programs , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[21]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[22]  K. S. How Tai Wah,et al.  An analysis of the coupling effect I: single test data , 2003, Sci. Comput. Program..

[23]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[24]  S. Inglis,et al.  Jumble Java Byte Code to Measure the Effectiveness of Unit Tests , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[25]  Yves Le Traon,et al.  AjMutator: A Tool for the Mutation Analysis of AspectJ Pointcut Descriptors , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[26]  Fabiano Cutigi Ferrari,et al.  Mutation Testing for Aspect-Oriented Programs , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[27]  L. Darrell Whitley,et al.  HOMAJ: A Tool for Higher Order Mutation Testing in AspectJ and Java , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[28]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[29]  James A. Jones,et al.  Fault interaction and its repercussions , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[30]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[31]  David Notkin,et al.  Detecting Redundant Unit Tests for AspectJ Programs , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[32]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[33]  Dewayne E. Perry,et al.  Toward understanding the rhetoric of small source code changes , 2005, IEEE Transactions on Software Engineering.

[34]  Rajib Mall,et al.  Dynamic Slicing of Aspect-Oriented Programs , 2008, Informatica.

[35]  Roger T. Alexander,et al.  A Candidate Fault Model for AspectJ Pointcuts , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[36]  Lech Madeyski,et al.  Searching for Strongly Subsuming Higher Order Mutants by Applying Multi-objective Optimization Algorithm , 2015, ICCSAMA.

[37]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[38]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[39]  Alex Groce,et al.  Mutations: How Close are they to Real Faults? , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[40]  Roland H. Untch.  Mutation-based software testing using program schemata , 1992, ACM Southeast Regional Conference.

[41]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[42]  Sudipto Ghosh,et al.  An approach and tool for measurement of state variable based data-flow test coverage for aspect-oriented programs , 2015, Inf. Softw. Technol..

[43]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[44]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.