A new virtual horizon

James Whittaker points to virtualisation one of the key things that will emerge in the future of testing and in my mind the Hyper-V revolution is definitely shaping up. At work here, we're already planning on ways to cope with private and shared VM "whole deployment" test environments as we iterate through the development cycles. Perhaps this is old-ish news to teams in product-focussed businesses. For Intergen as an IT service provider working in this space, with our small (days to weeks) test windows and many concurrent projects, this is huge.

I am also about to start execution of performance tests against a P to V (production to virtualised) environment - I remember talking to a client 18 months ago about how performance testing against a Virtualised environment should be treated as purely indicative, as the capability of VM's under load just didn't really approach that of real servers. Perhaps this is still true under stress, but the difference here is that we're talking about achieving acceptable "production-ready" response, and throughput, with only the CPU utilisation explicitly excluded from the set of acceptance criteria we'd use against the base metal. Funny how quickly things change. In any case, no doubt we'll learn something new.

I have a weather eye on the emergence of cloud computing Windows Azure / Amazon's EC3. Presuming that the application scales at all, I predict that these services will make it possible to scale out an app even if it is inefficient, just by buying more capacity. While you could worry that it threatens the 'Performance testing' discipline in some ways, I predict that it will make the justification for measurement and tuning more direct.

In the old days if it your app works acceptably well on the machines you already had (with some headspace) that was all you cared about. With the cloud, however, there is no fixed asset, so any performance improvements will save you operational cost pretty much immediately, and there will probably need to be a more regular assessment of your needs, especially if you have variability in your weekly / monthly / yearly demand.

In any case, this whole area is going to be new frontier, with a myriad of concerns not only technically but morally as well (e.g. legal jurisdiction, security/privacy). Exciting!

Feel free to share your thoughts on how VM technology is going to affect the tester.

Visual Studio Web Tests - Random Access is not quite Random

I've just been grappling with an issue in VS2008 Tester Edition that was skewing the load profile I'm after for my latest performance test assignment.

In a WebTest (either coded or UI-friendly) if you set "Random" access on Data Source and set the Number of Test Runs to a "Fixed Run count", the test runner will randomly select from all but the last row of data.
It works fine for 1 row, but with 2 or more rows, it will always ignore the last row. There is an n-1 bug in there somewhere....I've reported it to MS via the forums.

I've checked and VS2005 seems to have the same issue, and it affects at least csv and xml data source types.

The workaround is easy, as all you need is a padding row, but if you're wondering why the maths is off when using random access, this might be why.

The MSFT Web / Load Test Gurus

Before I forget, here are the blogs for the Microsoft gurus on the VS2005 tester toolset
Josh Christie - Bill Barnett and Ed Glas.

Also:

Sean Lumley and

Team blog

VS2005 Load Testing

Ok, so the ACT testing didn't go so great. Basically the Testing window was 4 hours (from 3p.m. to 7p.m.) and there were some significant responsivity issues I saw during that period. Anecdotally we saw a huge improvement after changing (a seemingly un-related) registry setting regarding the Active Directory implementation, but who knows? (we didn't re-test...)

I've been asked to look at the performance for another client and have spent quite some time with the VS2005 Beta 2 and getting a series of tests automated. It is a huge step up from the ACT tool although grappling with the most beta aspects is still a challenge.

Things I really like:

• The web test recorder works with HTTP 1.1, SSL and NTLM etc...
• The automatic handling of VIEWSTATE (thankyou, thankyou, thankyou)
• Out of the box extraction / validation rules and their extensibility (very powerful).
• Working in an Intellisense IDE.
• Visual representation of the requests, querystrings, form posts etc

I'll just quickly list some of the areas I've had some pain - if there is anyone out there who needs more info about any of these, i'm happy to share....

1. XML data binding - Nope not supported just yet as far as I can tell - use an SQL table if it's required.
2. Web /Load Test renaming and moving - also works inconsistently. I get warnings and misnamed files quite regularly when doing these prety basic operations. I've learned to be careful about this and usually any problems are obvious immediately. I just kill the copy and re-try until the tool gets it right (which is does eventually).
3. Load Test Results - Don't seem to be saved in the same way as other test type results are. When I end a test session and shutdown VS2005, the result view basically disappears. I suspect that there is something more I could do if we had the ability to publish the results to the Foundation Reporting service, but I'm not sure. The backend data can be pulled from the DB still though, but the front end presentation is so good. What I'm intending to do to work-around this is run the tests in VS2005 but have perf mon record the counters separately.
4. Test Result DB connectivity - This required some in-depth research to get running. There is information out there around this (http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=55303) but the conection string dialog is still flakey. I did have to re-install the tool to get back at the appropriate setting when I wanted to re-set it.
5. Random crashes - these happen a lot with test creation and de-bugging -although I must admit that I haven't lost data as a result, and I haven't had to abort a load test run in this manner either....When running web tests it is common to get the system "freezing".
6. The API is largely undocumented as yet- a pain if you want to do anything beyond what MSFT have provided for.
   

Start at Intergen and perfomance testing

This is my first post since joining Intergen. A pretty scary first week just trying to get to grips with the what and how of the projects and company ... seems like a good bunch of people, though.

I'm going to have to dust off the VBScript skills for the perfomance testing I'm doing with Damon. The ACT tool has some serious shortcomings in it's 'development environment' which will have the side effect of making me think HARD. This will be kind of good because it will drum in some understanding, but will probably take longer than the PM would like. In any case, it's nice to have some under the hood work to do and is precisely the kind of thing that MoE just didn't get near.

A couple of things that DC mentioned that seemed important so I will note them here while I remember.

1. DC mentioned that the is a machine config setting that needs to be watched on the webserver under test. I think in the web server machine.config file: the enableViewStateMac="false"...
2. The Client machine needs to be quite good for even emulating 100 concurrent users. As the spec is calling for 2400 for extreme load this will be interesting to try on anything that isn't massive.
3. I'll need to ensure I get permissions on the target web and db servers so I have access to the performance metrics.
   

Here is an interesting white paper from AK uni, which have yet to digest... and an interesting idea for a converted development framework.