Systematic Testing of Database Engines Using a Relational Constraint Solver

We describe an automated approach for systematic black-box testing of database management systems (DBMS) using a relational constraint solver. We reduce the problem of automated database testing into generating three artifacts: (1) SQL queries for testing, (2) meaningful input data to populate test databases, and (3) expected results of executing the queries on the generated data. We leverage our previous work on ADUSA and the Automated SQL Query Generator to form high-quality test suites for testing DBMS engines. This paper presents a detailed description of our framework for Automated SQL Query Generation using the Alloy tool-set, and experimental results of testing database engines using our framework. We show how the main SQL grammar constraints can be solved by translating them to Alloy constraints to generate semantically and syntactically correct SQL queries. We also present experimental results of combining ADUSA and the Automated SQL Query Generator, and applying our framework to test the Oracle 11g database. Our framework generated 5 new queries, which reveal erroneous behavior of Oracle 11g.

[1]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[2]  Carsten Binnig,et al.  Reverse Query Processing , 2007, 2007 IEEE 23rd International Conference on Data Engineering.

[3]  Carsten Binnig,et al.  QAGen: generating query-aware test databases , 2007, SIGMOD '07.

[4]  Yuetang Deng,et al.  Testing database transactions with AGENDA , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[5]  Rico Wind,et al.  Simple and realistic data generation , 2006, VLDB.

[6]  Suzanne M. Embury,et al.  An intensional approach to the specification of test cases for database applications , 2006, ICSE '06.

[7]  Meikel Pöss,et al.  Generating Thousand Benchmark Queries in Seconds , 2004, VLDB.

[8]  Margus Veanes,et al.  Qex: Symbolic SQL Query Explorer , 2010, LPAR.

[9]  Surajit Chaudhuri,et al.  Generating Queries with Cardinality Constraints for DBMS Testing , 2006, IEEE Transactions on Knowledge and Data Engineering.

[10]  Yannis Smaragdakis,et al.  Scalable automatic test data generation from modeling diagrams , 2007, ASE.

[11]  Donald R. Slutz,et al.  Massive Stochastic Testing of SQL , 1998, VLDB.

[12]  Rupak Majumdar,et al.  Dynamic test input generation for database applications , 2007, ISSTA '07.

[13]  Sarfraz Khurshid,et al.  Query-Aware Test Generation Using a Relational Constraint Solver , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[14]  Surajit Chaudhuri,et al.  Flexible Database Generators , 2005, VLDB.

[15]  Daniel Jackson,et al.  Finding bugs with a constraint solver , 2000, ISSTA '00.

[16]  Emina Torlak,et al.  Kodkod: A Relational Model Finder , 2007, TACAS.

[17]  Sarfraz Khurshid,et al.  Automated SQL query generation for systematic testing of database engines , 2010, ASE.

[18]  Daniel Jackson,et al.  Alloy: a lightweight object modelling notation , 2002, TSEM.