August 29, 2008

Pairwise Data-Driven Automation - Post Script

A caveat at this point: James Bach and Patrick Schroeder are of the opinion that pairwise testing should be approached with eyes open, and understanding the deficiencies of the approach, suggesting that random testing may be just as good. I've read their excellent article and I think I grok what they're saying.

They propose that the effectiveness of pairwise testing depends on 7 factors:
1) The actual interdependencies among variables in the product under test.
2) The probability of any given combination of variables occurring in the field.
3) The severity of any given problem that may be triggered by a particular combination of
variables.
4) The particular variables you decide to combine.
5) The particular values of each variable you decide to test with.
6) The combinations of values you actually test.
7) Your ability to detect a problem if it occurs.

The features of PICT address some of these points - not on it's own, but alongside robust analysis of the requirements and correctly implementing a model that follows them. The advantage it gives us in automation is that the model is external to the test (and therefore maintainable) and it generates the dataset rather than requiring a static source.

Anyway I've not seen anything else like this approach - I'd be interested to hear thoughts about whether you think it's useful.

Future thoughts - I could dynamically generate some wrapper code to refer to the model elements, this would give me build errors if the expected paramenters weren't present in the source, and do the tilda stipping or any other input polishing outside the test.