VS2010 Testing Features - Part 1 Camano

So two of our "strategy and innovation" team represented Intergen up at the Microsoft Professional Developers Conference in Los Angeles last week. They brought back a hard drive packed with goodies including the new Windows 7 OS and the latest CTP for VisualStudio 2010. I was super exited to get the Virtual Machine running today and play with the new testing features in VS2010 - there are several massive steps forward for QA pros (...and developers that care).

We're a Microsoft shop. I've been using the MS VS tools since 2005 for web / load test projects and using the unit test types for hooking into WatiN and other frameworks.What has been lacking has been support for planning tests and managing the results. Camano is the intended solution and it's a standalone app that runs against a TFS repository, with Test artifacts all managed as workitems.

The first step (and most important) is to get those document-based test plans/ test cases into a centralised tool that allows:

• progress tracking
• run reporting
• bug / issue administration
• It will also allow requirements and user story tracebility, if you go as far as managing these in TFS - which you should. (i.e. I can see you need not .. but you still should)

By recording a manual test you can generate a number of artifacts including a video (to assist with communicating the activities that uncover bugs) and an "automation strip" for playback and also to act as a basis of generating a true automated test. .

Playing with the features to "automate a manual test" reveals some of the more alpha quality areas, it just doesn't seem baked, although having said that the 1 GB restriction on my VM was not helping things (grindingly sloooow). I haven't quite grasped the Camano mechanism for running an automated test - perhaps that is just not there yet.

To be honest though, these particular features are icing, and I'll live through some pain in that area to get the managment stuff I need. I also understand that the CTP was cut in July - so they are 3 or 4 months on from this at Redmond.

Where things aren't quite there in Camano it does appear that TFS itself is further along. For example the reporting features and linking a test case with an automated test works in TFS but not in Camano. The only real downside as I see it is the need for a Team Foundation Server and the associated maintenance / cost overhead. I could see that being a battle for test teams that would otherwise win big out of going in this direction.

While there are some complexities in learning how this works, it offers a powerful features. There are also some promises of "test environment" VM management in the Team Lab SKU.
Comparing to the Rational / Mercury players isn't easy but for the price point (ESPECIALLY if your org uses VS / TFS already) its likely to come out swinging...

Learning and Understanding

I've just read this sample chapter from Andy Hunt's new book: Pragmatic Thinking and Learning: Refactor Your "Wetware", that deals with the Dreyfus model of skill acquisition. While this kind of introspection is probably more the domain of Michael (the Braidy Tester) Hunter's blog, I found it fascinating.
In particular it explained two things for me:

1. That Agile Projects really do need those really skilled practioners driving them in order to succeed. These are the Jedi Masters that just "feel the force" rather than "read the manual".
2. That a lot of the anxiety I feel when learning something new comes from that fact that I don't yet have that fully-formed picture of the conceptual framework. I can almost feel that discomfort lessen each time I get one of those "a-ha" moments. Which interestingly occur not when something I'm trying works, but rather, when I understand why all the previous attempts didn't.

I feel that this second point is important for testers to tune into - the better your internal model of the business/ techological processes, the less likely you are to miss important bugs in business/technological logic.

I'm sure others will relate to this idea, let me know if you "feel it" too. Anyway I can't wait to digest the rest of the book.

Visual Studio Web Tests - Random Access is not quite Random

I've just been grappling with an issue in VS2008 Tester Edition that was skewing the load profile I'm after for my latest performance test assignment.

In a WebTest (either coded or UI-friendly) if you set "Random" access on Data Source and set the Number of Test Runs to a "Fixed Run count", the test runner will randomly select from all but the last row of data.
It works fine for 1 row, but with 2 or more rows, it will always ignore the last row. There is an n-1 bug in there somewhere....I've reported it to MS via the forums.

I've checked and VS2005 seems to have the same issue, and it affects at least csv and xml data source types.

The workaround is easy, as all you need is a padding row, but if you're wondering why the maths is off when using random access, this might be why.

Pairwise Data-Driven Automation - Post Script

A caveat at this point: James Bach and Patrick Schroeder are of the opinion that pairwise testing should be approached with eyes open, and understanding the deficiencies of the approach, suggesting that random testing may be just as good. I've read their excellent article and I think I grok what they're saying.

They propose that the effectiveness of pairwise testing depends on 7 factors:
1) The actual interdependencies among variables in the product under test.
2) The probability of any given combination of variables occurring in the field.
3) The severity of any given problem that may be triggered by a particular combination of
variables.
4) The particular variables you decide to combine.
5) The particular values of each variable you decide to test with.
6) The combinations of values you actually test.
7) Your ability to detect a problem if it occurs.

The features of PICT address some of these points - not on it's own, but alongside robust analysis of the requirements and correctly implementing a model that follows them. The advantage it gives us in automation is that the model is external to the test (and therefore maintainable) and it generates the dataset rather than requiring a static source.

Anyway I've not seen anything else like this approach - I'd be interested to hear thoughts about whether you think it's useful.

Future thoughts - I could dynamically generate some wrapper code to refer to the model elements, this would give me build errors if the expected paramenters weren't present in the source, and do the tilda stipping or any other input polishing outside the test.

Pairwise Data-Driven Automation - Part 3

So at what we hope to achieve in this post is
1. Get a WatiN test going under the Visual Studio test framework
2. Hook the appropriate PICT output up as a data source for the test
3. Run the the test and see many inputs and many results running and passing.


1. First specify a data source from the Pict output file. You can also use the app.config to hook this up if you wish.
i.e.

   [TestClass]
    public class UnitTest1
    {
     
       [DataSource("System.Data.OleDb", 
       "Provider=Microsoft.Jet.OLEDB.4.0;
       Data Source=.;Extended Properties='text; 
       FMT=TabDelimited;HDR=YES'", 
       "frombat#txt", 
       DataAccessMethod.Sequential), 
       TestMethod]

   }

One gotcha here, you'll need to include the model input file and schema.ini (for tab delimted text files) in the test deployment settings
( Test > Edit Test Run Configurations > Deployment )

2. I'll assume you have some familiarity - with WatiN, so build a test method that hits our login page and wire in the parameters from the Pict output.

For our purposes we'll want to supply a bad username / password and verify that the right error message is returned.
Something like this...another gotcha - the tildes need to be stripped (I've left that in....)

public void MainSiteIsUp()
        {
            using (IE ie = new IE("http://www.actionthis.com"))
            {
                // Load variables from the file
                    
                    string email = context.DataRow["EMAIL"].ToString();
                    //strip the leading tilde if there is one...
                    if (email.StartsWith("~")) email = email.Remove(0, 1);

                    string password 
                       = context.DataRow["PASSWORD"].ToString();
                    string message 
                       = context.DataRow["$RESULT"].ToString();

                LoginPage.Login(ie, email, password);
                Assert.IsTrue(ie.Text.Contains(message));
            }

3. Select and run the test. It should build, cycle through the 7 combinations and check for the various error messages - all green!

Yay - big pat on the back!

Pairwise Data-Driven Automation - Part 2

In this post I'm going to use the PICT model to define the "unsuccessful login" test cases for our application.

I'll freely admit that the example I use here is probably not the strongest application of PICT, being relatively straightforward in the number of variants and potential cases it can output (i.e. I can keep all these rules in my head an construct a small number of tests to cover all of the variants anyway). Where it really comes into it's own is when there are more complex interactions and constraints that blow out the number of possible test combinations. I started looking at using it for Credit Card validation rules within the app, where there are a lot more rules regarding the minimum data requirements and the validation rules. But that's all internal logic, and the login screen anyone can see - so lets play with that.

Let's start by defining the rules for the login screen.

To login you need a valid email address, a password and an account in the system with those credentials. The system gives one of three error messages different error messages

...depending on the rules we'll specify in the model file.

Read up about the way that pict works in the user guide (comes with the download) or the article i referred to last post. Basically the above "spec" translates into a ModelFile like this, with 2 input parameters and an output parameter. I have taken a minimal set of invalid email address data to illustrate the point (i.e. there are others I'd include here if I were being throrough).

#
# Login To ActionThis
#

EMAIL: nodomain,,@domain.only, onepart@domain,correct@format.em.ail
PASSWORD: password,
$RESULT: The Email Address entered does not appear to be valid., Email Address and Password may not be blank., Login failed. Please check your username and password and try again.

# Blank Email or Password rule
IF [EMAIL] = "" OR [PASSWORD] = ""
THEN [$RESULT] = "Email Address and Password may not be blank.";

# Invalid Email address rule
IF [EMAIL] IN {"nodomain", "@domain.only", "onepart@domain" }
THEN [$RESULT] = "The Email Address entered does not appear to be valid.";

# Valid email / password but no account rule
IF [EMAIL] = "correct@format.em.ail" AND [PASSWORD] = "password"
THEN [$RESULT] = "Login failed. Please check your username and password and try again.";

Assuming you have pict installed, you can run pict directly from the commandline as : pict.exe ModelFile.txt > Outputfile.txt (try it and see). The output file is a tab delimted text file, which I'm intending to feed into the automation.

So lets get a new C# test project happening in VS2008. I'll plumb the PICT in as a pre-build event for the project, so that any updates in the Modelfile are pulled through for the test run directly as and when that happens.

One gotcha here is that running the pict.exe anywhere under the local solution directory gives me 9009 errors. Not sure why, possibly a PATH conflict? I was forced to running the pict.exe from it's native location (i.e. in C:\Program Files\PICT\) but referencing the files in the project folder. anyway, the pre-build event command line looks like :

cd \"Program Files"\PICT\
pict.exe "$(ProjectDir)ModelFile.txt" > "$(ProjectDir)pictoutput.txt"

OK so if your input file exists in the project dir, that should actually build and output a tab file, pictoutput.txt.

Next up we'll wire it up to WatiN via a unit test in VisualStudio....

Pairwise Data-Driven Automation

I've spent a bit of time in the last few days getting the WATIN Framework going with called PICT (the Pairwise Independent Combinatorial Testing tool), for pairwise test case generation.

To date we've used hardcoded data values for most automated tests, and this leads to either embedded data all over the show (smelly!) and/or multiple calls to the execution logic handled within the test (inelegant)...and of course the coverage could be better.

My first step towards a data-driven test used a "blunt force" attack on the data inputs (i.e an all-pairs approach, varying one input at a time) which generated a phenomenal number of test cases. As it was, our suite took quite some time to complete, but this approach just made it insane - anything that reduces execution time without compromising the new juicy coverage was going to be awesome.

That's where PICT comes in. It's developed by a couple of engineers at Microsoft and is used internally by them top generate test cases "pairwise" in order to get the Input combination level most likely to find issues. You can read about the tool here and download v3.3 from here. Basically it offers a way to specify a set of outputs, their dependencies or constraints if any and also the expected results of those input(s). This gives us the ability to specify the input/output rules in one place (the PICT Input file) and get an output set of test cases optimised for coverage without redundancy.

So where I'll get to over the next few posts is a rather trivial but nonetheless working sample of PICT / WATIN working together to give us data-driven automation, hopefully in a maintainable way. There were some issues that I've found I've had to work around on the way, so I'll let you know about those too!

Next Up: I'll Dive in to the PICT Mapping Rules for our example.

Introducing ActionThis

Here's why I've been so quiet of late - work has been pretty absorbing. I'm part of the team working on an amazing new Web2.0-ey project called ActionThis.

This is an application focussed on project execution and recovery - it's presenting exciting new Testing challenges partly because it represents a paradigm shift in it's approach - and partly because its the first truly agile software project I've been involved with.

At Intergen we've now entered into a partnership arrangement with CompuWare. I'm using their TestPartner and QADirector in the day to day work on Action This. I expect to be presenting a Twilight session shortly on this once we hit our Beta Release. I expect this to take the form of a Case study about the ActionThis QA process, automated testing and AJAX apps.

It's an awesome thing to be a part of, and I can't wait to tell you more (when I'm allowed to), but you can sign up for more news as it happens at >> www.actionthis.com

5 Things

I got tagged by Gabe in the 5 things blog meme.

I normally wouldn't do this kind of thing, but anyway, here goes:

1. Tip-Top Pineapple Crush Icecream gives me hives. The world is now a safer place now they've stopped making it.

2. I was in Rarotonga during Hurricane Sally (1 Jan 1987). The eye of the storm went over the top of us at halfway through. Apparantly this is relatively rare, but it gave us a chance to secure the section of roof which had lifted off. I was shit-scared.

3. When I first came to Wellington I stayed in a flat full of Anarchists. Lots of mohawks, dogs and a dessicated pizza hanging from the living room ceiling. One thing that still makes me retch to this day was the fridge that the dog food was kept in. I'd guess it hadn't been cleaned in 5 years and their power was disconnected regularly. That relationship didn't last. Absolutely. Disgusting.

4. After graduation, I got a job at Victoria University which involved (amongst other things) running and maintaining a Particle Accelerator for the Nuclear Physics Group.

5. On my OE I spent months in Scotland without work. I could only afford to eat bread, jam and porridge, no heating and no fun. ( Actually looking back on it now it seems like a bizarre cultural immersion programme.)

I agree with Mr Peters - Everyone has already done this, so I won't spread the madness.

Watin / WatiNFixture - New Versions

WatiN 0.9.5 was relased 1/1/'07 with
WatiNFixture 0.2 following soon after at:

I've now got a fairly good handle on a WatiN test pattern that works for me - basically a modified form of a pattern blogged by Richard Griffin. He's the guy that spooled up the WatiN Recorder tool to generate WatiN script from a browser session..

My favourite new feature in WatiN 0.9.5 is the ability to filter out a collection of elements - this is particularly useful in repeater sections where an item of interest may be buried in a list with little to distinguish it from other items. I had previously been doing this by reg-ex matching within the HTML body and trying to strip out the relevant button ids etc.

Much, much cleaner.

WatiN and WatiNFixture

I'm sure some of you will have heard of WatiR (Web application testing in Ruby).

We'll I've recently discovered a .NET offshoot of the same (called WatiN), which suits:

• my limited programming ability by virtue of being C# scripted (although any .NET language will work).


• my .NET FitNesse environment

I've played around somewhat with using WatiN browser automation via VS2005 unit tests, and this works really well.
I also tried calling it via FitNesse through ActionFixtures and the like. It was cool, but a little heavy on reliance on the fixture programming.

Anyway in trawling the FitNesse forums ( http://tech.groups.yahoo.com/group/fitnesse/ ), I stumbled across Jeff Parker's work on a dedicated FitNesse Fixture for WatiN automation, (un-surprisingly called WatiNFixture). This allows you to create an IE instance and control the actions and validation is a key-word manner. I hit him up for the code but it was getting some polishing and was likely to come out with WatiN 9.0 :-(

WatiN release 9.0 was put out yesterday (here) sans WatiNFixture. Things are really coming together, though - Frame / iFrame support and Dialog Box handling is all in this release. WatinFixture 0.1 also launched yesterday (here) so it looks like they're collaborating rather than integrating at this stage.

Anyway if you're into test / IE automation and Fit or FitNesse check them out.

FitNesse and Data Conversion

I blogged quite some time ago about some data conversion testing I was doing. The customer has a freaky proto-database made up of hundereds of separate MS Excel worksheets and frequently running into the old Excel maximum row count limit. We're doing the sensible thing and getting it into a relational DB (in this case MSSQL 2005). Dont ask me how they managed up until now.

I'm way overdue for an update on this, so here we go:

I ended up working mostly with FitNesse and SQL stored procedures.

I absolutely love FitNesse. For those of you who dont know it, FitNesse is based on Fit - a tool for writing and executing 'story tests' that both describe the desired behaviour of the system (requirements) and validate that the behaviour is achieved. FitNesse is a wiki based platform for writing and running these Fit tests. You can read more about it here.

I had (the .NET 2.0 variant of) Fitnesse executing stored procedures retrieving data for given set of parameters, and checking that the result sets matched. The tests can involve many assertions, and they can be organised into Suites covering logical areas and suites of suites - covering pretty much the whole system.

Given that there is absolutely masses of data (and conversion rules) I wanted to automate as much of these tests as possible, this meant I could keep working through the spec (chasing the dev) but every time I ran a test, I'd re-run ALL the old ones. Thing would break way back quite regularly. Once we got to feature complete, we just satrting doing bugfixes, and turning red tests into green ones.

At this point (near the end I'm hoping) I have 2300 odd tests running with every migration pass. Of course I could have had some big arsed SQL scripts going and doing the same thing, but for me the beauty of the FitNesse tool is the following two points:

Reporting - The outcome of the test runs is very obvious ( green / red colour coded to the test and assertion that has failed)

Maintainability - The tests themselves (including the expected results) are easily editable.

Oh ... and as they're "self documenting" pretty much circumvent writing test cases in Word.

Im sold on it.

Now I'm looking at getting some browser automation going with FitNesse. I'm looking at WatiN ( an I.E. automation framework like the Ruby WATIR framework, but for .NET ). More on this later.

A disclaimer

Jerms is about to circulate a list of Intergen bloggers.

It's probably appropriate that I say at this point that this blog is probably never going to be the central authority on Test Tools, Microsoft Test Tools or even Piers' dodgy hacked test tools.

Even if I wanted it to be.

I use it for collecting ideas and saving them where I can get at them later.

...you may or may not be interested in going through my junk.

The MSFT Web / Load Test Gurus

Before I forget, here are the blogs for the Microsoft gurus on the VS2005 tester toolset
Josh Christie - Bill Barnett and Ed Glas.

Also:

Sean Lumley and

Team blog

MAF conversion - update 1

So the MAF stuff:

At the moment the system puts some data in to a database, and writes to an error log where it finds any issues. I haven't looked at the relationships in the target system or even the data quantity yet, but first things first....

I scrounged this stored proc, which will count the number of rows in each table after the migration run has occured. At present these are stored in a temporary table and the results just written to the SQL Query analyser, but I'm looking to keep a record of these (outside the database itself - so as not to change the beast I'm checking).

I stumbled across the bcp utility and the xp_cmdshell proc, which will let me write the results out to a text file of my choice - still feels a little inelegant, but I think it'll do for now.

MAF - Data wars

Here we go. I'm on MAF with the Featherston Street team. Prob'ly be there in the new year.

Weird kind of testing this Data Migration - Building an application you hope will be run as few times as possible. I'm guessing that because there is no UI (and because Mr Andrew Peters is on-board) I'll be encouraged to implement a Fitnesse suite.

Check out Fitnesse here.

VS2005 Web test custom validation rule

/*********************************************************/

using System;

using System.Text;

using System.ComponentModel;

using System.Collections.Generic;

using System.Collections.Specialized;

using Microsoft.VisualStudio.QualityTools.WebTestFramework;

using Microsoft.VisualStudio.QualityTools.WebTestFramework.Rules;

/**********************************************************************************************/

// Handle validation of response header values

/**********************************************************************************************/

namespace HeaderValidationRule

{

/**********************************************************************************************/

// Inherits validation rule

/**********************************************************************************************/

public class ValidateHeader : ValidationRule

{

// Header name

private string m_headerStringParameterName;

public string HeaderStringParameterName

{

set

{

m_headerStringParameterName = value;

}

get

{

return m_headerStringParameterName;

}

}

// Header value

private string m_headerStringParameterExpectedValue;

public string HeaderStringParameterExpectedValue

{

get

{

return m_headerStringParameterExpectedValue;

}

set

{

m_headerStringParameterExpectedValue = value;

}

}

// Fail if not found?

private bool m_isRequired = true;

[DisplayName("Is Required")]

public bool IsRequired

{

get

{

return m_isRequired;

}

set

{

m_isRequired = value;

}

}

/**********************************************************************************************/

// Add logic to handle header validation

/**********************************************************************************************/

public override void Validate(object sender, ValidationEventArgs e)

{

e.IsValid = false;

e.Message = "Not found";

for(int i = 0; i < e.Response.Headers.Keys.Count; ++i)

{

if (e.Response.Headers.GetKey(i) == HeaderStringParameterName)

{

string strVal = e.Response.Headers.Get(HeaderStringParameterName);

if (strVal.Contains(HeaderStringParameterExpectedValue))

{

e.IsValid = true;

e.Message = "Found";

break;

}

}

}

}

}

}

Binding CSV data to a web test in VS2005

(nicked from the VSTS Test Tools Forum)

To add a csv file as a datasource, do the following:
1) Create a csv file that looks like the following:
username,password
user1,password1
user2,password2

The first row is for column headers.

2) Click the add datasource button for the webtest
3) Choose Microsoft Jet 4.0 OLE DB Provider as the OLE DB Provider
4) Click the Datalinks button
5) On the connection tab, enter the directory that the csv file is in for the "Select or enter a database name:" text box. Enter just the directory.
6) Click on the all tab.
7) Double Click Extended Properties
8) Enter text and hit OK
9) Click Ok for the Data Link Properties dialog
10) Click Ok for the connection properties dialog
11) Choose the csv file in the choose tables dialog.
12) Add the datasource to the field you want to bind to. The column headers in the csv file will be used for the field names.

The Perfect Bug

(nicked from a VSTS Quality Tools blog) Whether we’re running automated or manual test cases, whenever we come across a failure it is often a good idea to log the issue in the bug database. In the Test Results window after a test run, you can use the list of failures to investigate a failure, rerun a test under the debugger, and finally associate the failure with a work item.

When you activate this feature, the product does a little bit of the work for you by opening a new bug form and filling in some of the fields.

If you have access to a Team Foundation Server, I recommend trying this out. First execute a test case that will fail, publish it, and then execute the “Create Work Item” menu item off of the context menu from the failed case.

[TestMethod]

public void TestMethod1()

{

Assert.Fail("Misc. bug in this code");

}

What you’ll see is a new bug form open with a bug title prefix (in my case “TestMethod1: “). You’ll also see in the Comment and History section the error message text (in my case “Assert.Fail failed. Misc. bug in this code”).

Note: Alternatively, if the bug you want to associate this failure with already exists, you can execute another menu item: Add to Work Item. This helps you add failure information from this case to an existing bug.

The rest, and arguably the real value, comes from you – we’ve just tried to automate some of the process that slows you down.

So, what do you put into the bug? Another way to ask this question is: what will the developer need to see in order to fix the bug in a very efficient manner? What information will increase the likelihood of a bug fix? How can I reduce the number of bugs that get resolved as Not Repro? A process that helps you enter bugs that achieve all those things could be called the Perfect Bug.

The Perfect Bug is a good thing to achieve. Others will more quickly understand it. Management will make quicker, but more informed and accurate decisions for the product. The developer will be less likely to misinterpret the bug and provide the wrong fix. You have supplied the developer with critical information that makes fixing the issue as quick and painless as is possible. Everyone will spend less time on the bug (reading, comprehending, etc).

The absolute perfect bug is not usually attainable. There is obviously a limit to the amount of time you should put into a bug. There has to be a corresponding benefit to the time you put into it. However, we can first focus on entering really good bugs and work our way up.

A high quality bug is easy to read. It is concise, yet contains additional crucial information. How can we communicate so concisely and clearly?

Some of these items are specific rules to remember, but in general it can be wrapped up with a set of principles:

• Make it concise and easy to parse.
• Considerably large data that would break the previous principle can be included elsewhere and simply referred to.
• It should be easy to find by another person (tester looking for duplicate, others looking for bug they once saw).
• The bug entry should make the best and most accurate case for fixing.

A perfect bug has…

An accurate and concise title

• A bug title should not be too generic. The reader won’t understand what the bug really is. Bad example: App doesn’t work. What app? How doesn’t it work? What is specific about this scenario that causes it to happen?
• A bug title should not be too long. The longer the title is, the more the reader has to concentrate; they may have to reread it several times. Bad example: Leave defaults in a new test and run it. Test outcome is "Failed". Should it be "Error" or "Not Runnable" instead? This really could be said in a lot fewer words. Most of that title belongs in the Repro Steps section. Try Defaults for new test yields a ‘Failed’ result.
• A bug title should include relevant error messages or crash address. This makes it easier for others who are searching for duplicates. Good example: FileNotFound Exception in ObjectStore.css line 47 when opening file with .xxx extension. If I get this error when testing and search on it, this bug title will immediately pop out at me. Perfect!
• A bug title should all words spelled correctly, especially error messages. Take the time to make sure you spell words in your title correctly (do I hear an endorsement for built in spell check?). People looking for duplicates will not find yours and this causes more work for everyone.

Severity and Priority that are accurate

• Really think through the severity and priority you set for a bug. Consider other bugs you have entered and compare this bug to those. Know that developers (hopefully) use priority to set the order in which they fix bugs.
• If you enter a bug with what may seem like an unusual sev/pri or if the sev/pri are high, it would be very helpful to others if you explain your case for it. Especially do this if you change pri/sev.
• As a product group, define what a bug means to be sev 1 and so forth. The work item tracking solution has a feature to show a tool tip if you hover over the labels which can be used to reinforce the definitions. Here are some typical definitions used by Microsoft for your benefit:

Sev 1: Critical Failure. Completely breaks product or large set of features. Unusable. Significant risk or liability if release (security, legal).

Sev 2: Major Impact / Functionality Broken. Breaks major functionality contributes to overall instability in this area, non-fatal assertions. E.g., Statement Completion not active at all or memory leaks. Regression from prior release.

Sev 3: Minor Impact / Functionality Impaired. Breaks major functionality in a minor way or breaks minor functionality completely. E.g., missing item from list for statement completion.

Sev 4: Little / No user impact. Can still use product / features. Minor functionality problems or UI blemishes or other issues that do not impact customers use or perception.

Pri 0: “NOW” Bug. Work stoppage, no work around. Blocking further progress in area or by group. Fix Immediately! 24 hr turn around expected!

Pri 1: Showstopper. This is deeply impacting customers OR internal progress. Worthy of a Service Pack or QFE. Fix Soon! Also, required to fix for RTM.

Pri 2: Important Bug. Required fix for RTM. Can be fixed any time before RTM.

Pri 3: Something we would like to fix but not required to fix to ship the product.

Pri 4: An unimportant bug or request. A bug that will likely not be fixed.

One manager at MS puts bug priority in terms that might resonate better with you for priority:

Pri 1: Will slip the product indefinitely to get this in.

Pri 2: Will slip our date within limits to get this in. Painful cut if not in.

Pri 3: Will not even think about slipping the product for any of these.

If you are unsure about pri/sev, chat with others to level set your expectations.

If you notice that management or others regularly resets your bugs’ sev/pri, ask them to discuss why they see it differently.

Filled in fields (customize your bug form)

• These are all examples of custom fields we use at Microsoft. You can also add these to your own custom bug form.
• How Found

The idea is that the tester indicates how they found the bug, as in via what kind of testing. Was it a Test Pass? Automation? Ad hoc Testing? Customer Feedback? Bug Bash?

Your organization can use this field for metrics. If you know you found 30% of your bugs via automation, it’s a compelling reason for your organization to invest more heavily in it. If you find that bug bashes result in more bugs then you’ll know to plan more of those. You get the idea.

• Environment

Can be very helpful with reproducing a bug. Enter OS, proc (32/64-bit), product flavor.

• Blocking

Is this a blocking bug? You should definitely mark the bug as such and explain why in the description.

A blocking bug usually is around one of these:

1. Precludes a build from being generally testable
2. Prevents testing of other features in the area
3. Precludes a build from being generally safe for dogfooding
4. Breaks defined user scenario
5. Degrades an feature area quality bar so that it is not meeting expectations

Repro Steps

• You can add a large, multi-line pane next to Comment and History to hold the repro steps. Unlike Comment and History, this field can be updated at any time, whereas Comment and History can only be appended to.
• This is where you can make the biggest difference. Repro steps tell the reader a LOT about the bug. How easy to repro is this? Is the customer likely to run into it? Does it require the planets to align?
• Should include three sections: Repro Steps, Results, and Expected Results.
• If your actual steps to repro seem long, you will confuse the reader. They will also think the likelihood of the bug impacting most customers is slight. My suggestion is to include two repro steps. Have one be how most customers will hit it. The second one will be how to actually reproduce the bug from start to finish for a developer or another tester. Blurring the line between these two often masks how likely a bug is to be encountered. Make a good case for your bug to be fixed by describing the customer scenario separate from the straight steps to repro.

Extra info in the description

• Here is where you can put any other input you have. Your expertise is highly valued. Did you debug into the code a bit and find something relevant? Put that here!

Your bug may include a reference to the offending source. Excellent example: Incorrect permission looked up - uses UPDATE_DATA permission instead of PUBLISH_DATA. The description could say, “Note: This is because <path to file>\Service.cs:80 is requesting "UPDATE_DATA" rather than the publish permission.”

You could include a detailed description of the architectural flaws that led to the problem.

Do you have any root cause analysis? Do you have any proposed fixes and comments about pros/cons of each fix?

• Are there caveats to the bug that you think should be known? Is the problem easily avoided or resolved? Does the issue never go away? What about after reopening the solution or restarting the IDE?
• This is also where you can comment on your sev/pri rating. If this is a regression, here is where you should mark it. A regression is a set of repro steps that used to work, either in a previous release or previous build that no longer works. Management may be more likely to take a bug if quality has regressed.
• Is this issue a crashing bug?

If the OS has crashed (blue screen), do you have a Kernel Dump or a Kernel Mode Debugger attached to a repro machine?

UI crashing bugs should have a call-stack and mini-dump.

ICE (internal compiler error) bugs should have a preprocessed file. If a CHK build is available does the crash repro on CHK, and are there any Asserts hit? When in doubt, keep the debug session active and contact the owning dev.

• You could include a list of test cases to run to verify the fix. Oooh, ahhhh.
• Still very important… make all of the above readable. Format it in such a way that one can read through the bug and quickly identify relevant information.

Attached files

• There are several different kinds of files that may be very relevant. Perhaps the most helpful is a screenshot of UI that is bugged, especially in localization issues. Often you can say a lot more with a picture than you can accurately convey in text. Indicate that you’ve attached a picture so others know to look in the Files tab.

The picture should be JPG or PNG format. BMP and other formats are generally too big.

Crop the picture to what is needed to show.

If you have a lot of code to include in a repro step, it might be best to simply attach a file and refer to it in the bug Repro Steps.

VS2005 Load Testing

Ok, so the ACT testing didn't go so great. Basically the Testing window was 4 hours (from 3p.m. to 7p.m.) and there were some significant responsivity issues I saw during that period. Anecdotally we saw a huge improvement after changing (a seemingly un-related) registry setting regarding the Active Directory implementation, but who knows? (we didn't re-test...)

I've been asked to look at the performance for another client and have spent quite some time with the VS2005 Beta 2 and getting a series of tests automated. It is a huge step up from the ACT tool although grappling with the most beta aspects is still a challenge.

Things I really like:

• The web test recorder works with HTTP 1.1, SSL and NTLM etc...
• The automatic handling of VIEWSTATE (thankyou, thankyou, thankyou)
• Out of the box extraction / validation rules and their extensibility (very powerful).
• Working in an Intellisense IDE.
• Visual representation of the requests, querystrings, form posts etc

I'll just quickly list some of the areas I've had some pain - if there is anyone out there who needs more info about any of these, i'm happy to share....

1. XML data binding - Nope not supported just yet as far as I can tell - use an SQL table if it's required.
2. Web /Load Test renaming and moving - also works inconsistently. I get warnings and misnamed files quite regularly when doing these prety basic operations. I've learned to be careful about this and usually any problems are obvious immediately. I just kill the copy and re-try until the tool gets it right (which is does eventually).
3. Load Test Results - Don't seem to be saved in the same way as other test type results are. When I end a test session and shutdown VS2005, the result view basically disappears. I suspect that there is something more I could do if we had the ability to publish the results to the Foundation Reporting service, but I'm not sure. The backend data can be pulled from the DB still though, but the front end presentation is so good. What I'm intending to do to work-around this is run the tests in VS2005 but have perf mon record the counters separately.
4. Test Result DB connectivity - This required some in-depth research to get running. There is information out there around this (http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=55303) but the conection string dialog is still flakey. I did have to re-install the tool to get back at the appropriate setting when I wanted to re-set it.
5. Random crashes - these happen a lot with test creation and de-bugging -although I must admit that I haven't lost data as a result, and I haven't had to abort a load test run in this manner either....When running web tests it is common to get the system "freezing".
6. The API is largely undocumented as yet- a pain if you want to do anything beyond what MSFT have provided for.