June 20, 2008

Pairwise Data-Driven Automation

I've spent a bit of time in the last few days getting the WATIN Framework going with called PICT (the Pairwise Independent Combinatorial Testing tool), for pairwise test case generation.

To date we've used hardcoded data values for most automated tests, and this leads to either embedded data all over the show (smelly!) and/or multiple calls to the execution logic handled within the test (inelegant)...and of course the coverage could be better.

My first step towards a data-driven test used a "blunt force" attack on the data inputs (i.e an all-pairs approach, varying one input at a time) which generated a phenomenal number of test cases. As it was, our suite took quite some time to complete, but this approach just made it insane - anything that reduces execution time without compromising the new juicy coverage was going to be awesome.

That's where PICT comes in. It's developed by a couple of engineers at Microsoft and is used internally by them top generate test cases "pairwise" in order to get the Input combination level most likely to find issues. You can read about the tool here and download v3.3 from here. Basically it offers a way to specify a set of outputs, their dependencies or constraints if any and also the expected results of those input(s). This gives us the ability to specify the input/output rules in one place (the PICT Input file) and get an output set of test cases optimised for coverage without redundancy.

So where I'll get to over the next few posts is a rather trivial but nonetheless working sample of PICT / WATIN working together to give us data-driven automation, hopefully in a maintainable way. There were some issues that I've found I've had to work around on the way, so I'll let you know about those too!

Next Up: I'll Dive in to the PICT Mapping Rules for our example.

No comments: