Software Assurance by Bounded Exhaustive Testing

Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. A particularly interesting case of BET arises in the context of systems that take structurally complex inputs. Early research suggests that the BET approach can reveal faults in small systems with inputs of low structural complexity, but its potential utility for larger systems with more complex input structures remains unclear. We set out to test its utility on one such system. We used Alloy and TestEra to generate inputs to test the Galileo dynamic fault tree analysis tool, for which we already had both a formal specification of the input space and a test oracle. An initial attempt to generate inputs using a straightforward translation of our specification to Alloy did not work well. The generator failed to generate inputs to meaningful bounds. We developed an approach in which we factored the specification, used TestEra to generate abstract inputs based on one factor, and passed the results through a postprocessor that reincorporated information from the second factor. Using this technique, we were able to generate test inputs to meaningful bounds, and the inputs revealed nontrivial faults in the Galileo implementation, our specification, and our oracle. Our results suggest that BET, combined with specification abstraction and factoring techniques, could become a valuable addition to our verification toolkit and that further investigation is warranted.

[1]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[2]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[3]  Sarfraz Khurshid,et al.  Software assurance by bounded exhaustive testing , 2004, IEEE Transactions on Software Engineering.

[4]  Roberto J. Bayardo,et al.  Using CSP Look-Back Techniques to Solve Real-World SAT Instances , 1997, AAAI/IAAI.

[5]  David Coppit,et al.  Shared semantic domains for computational reliability engineering , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[6]  Sarfraz Khurshid,et al.  An Evaluation of Exhaustive Testing for Data Structures , 2003 .

[7]  Sarfraz Khurshid,et al.  Checking Java implementation of a naming architecture using Testera , 2001, Workshop on Software Model Checking @ CAV.

[8]  David Coppit,et al.  The Galileo fault tree analysis tool , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[9]  A. Jefferson Offutt,et al.  Generating Tests from UML Specifications , 1999, UML.

[10]  W E Vesely,et al.  Fault Tree Handbook , 1987 .

[11]  David Coppit,et al.  Developing a low-cost high-quality software tool for dynamic fault-tree analysis , 2000, IEEE Trans. Reliab..

[12]  Mark Allen Boyd.  Dynamic fault tree models: techniques for analysis of advanced fault tolerant computer systems , 1992 .

[13]  David Coppit,et al.  Sound methods and effective tools for engineering modeling and analysis , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[14]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[15]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[16]  David Coppit,et al.  Engineering modeling and analysis: sound methods and effective tools , 2003 .

[17]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[18]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[19]  David Coppit,et al.  Formal semantics of models for computational engineering: a case study on dynamic fault trees , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[20]  Daniel Jackson,et al.  Alcoa: the Alloy constraint analyzer , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[21]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[22]  Elaine J. Weyuker,et al.  Theories of Program Testing and the Application of Revealing Subdomains , 1980, IEEE Transactions on Software Engineering.

[23]  Mark Saaltink.  The Z/EVES System , 1997, ZUM.

[24]  Salvatore J. Bavuso,et al.  Dynamic fault-tree models for fault-tolerant computer systems , 1992 .

[25]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[26]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[27]  Sarfraz Khurshid,et al.  A Case for Efficient Solution Enumeration , 2003, SAT.

[28]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[29]  Thomas Santen,et al.  Automating Test Case Generation from Z Specifications with Isabelle , 1997, ZUM.

[30]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[31]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[32]  Nancy G. Leveson,et al.  An experimental evaluation of the assumption of independence in multiversion programming , 1986, IEEE Transactions on Software Engineering.

[33]  Sharad Malik,et al.  Chaff: engineering an efficient SAT solver , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[34]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[35]  J. Dugan,et al.  A modular approach for analyzing static and dynamic fault trees , 1997, Annual Reliability and Maintainability Symposium.

[36]  Hans-Martin Hörcher,et al.  Improving Software Tests Using Z Specifications , 1995, ZUM.

[37]  K. Sullivan,et al.  Galileo: a tool built from mass-market applications , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[38]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[39]  John C. Cherniavsky,et al.  Validation, Verification, and Testing of Computer Software , 1982, CSUR.

[40]  David A. Carrington,et al.  A Framework for Specification-Based Testing , 1996, IEEE Trans. Software Eng..

[41]  Elliot Schwartz,et al.  Design and implementation of intentional names , 1999 .

[42]  David Coppit,et al.  Formal Semantics for Computational Engineering: A Case Study on Dynamic Fault Trees , 2000 .

[43]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[44]  Daniel Jackson,et al.  Elements of style: analyzing a software design feature with a counterexample detector , 1996, ISSTA '96.

[45]  Joanne Bechta Dugan,et al.  Dependability assessment using binary decision diagrams (BDDs) , 1995, Twenty-Fifth International Symposium on Fault-Tolerant Computing. Digest of Papers.