tag:blogger.com,1999:blog-123522062024-02-07T15:35:35.046+13:00The Circus Powers of Pierzapin:: Taste-Testing Life ::Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comBlogger27125tag:blogger.com,1999:blog-12352206.post-4802990954436407462011-06-13T20:38:00.014+12:002011-06-15T15:33:52.091+12:00WELLINGTON SIGIST - June 15 2011<strong>This could have been the last slide</strong><br />Anders Dinsen's Blog :<a target="new" href="http://blog.asym.dk/2011/04/13/the-problem-with-test-documentation/">The problem with test documentation </a><br /><a target="new" href="http://www.mindgenius.com/">Mind Genius</a><br /><a target="new" href="http://wiki.meego.com/Quality">Use a Wiki for Test Doucmentation<br /></a><br /><strong>Analogue vs Digital</strong><br /><a target="new" href="http://wendyju.com/publications/thinking_erasable_DISTRO.doc.pdf">Whiteboard socialology study</a><br /><a target="new" href="http://thecoccinella.org/">Digital whiteboards</a><br /><a target="new" href="http://lizkeogh.com/2011/03/04/step-away-from-the-tools/">Conversations drive tests</a><br /><a target="new" href="http://www.youtube.com/results?search_query=rsa-animate&aq=0&oq=RSA-Animate">RSA-Animate series on YouTube</a><br /><br /><strong>Wax on, Wax off</strong><br /><a target="new" href="http://www.amazon.com/Testing-Software/lm/1MSK9Q9TDEGVK">Background Reading - The Test Classics on Amazon </a><br /><a target="new" href="http://technet.microsoft.com/en-us/scriptcenter/dd742419.aspx">Powershell</a><br /><a target="new" href="http://www.ruby-lang.org/en/">Ruby</a><br /><a target="new" href="http://www.python.org/">Python</a><br /><br /><strong>Computers pwn me at Data</strong><br /><a target="new" href="http://www.generatedata.com/">http://www.generatedata.com/</a><br /><a target="new" href="http://www.red-gate.com/products/sql-development/sql-data-generator/">Red Gate Data Generator</a><br /><a target="new" href="http://autofixture.codeplex.com">AutoFixture</a><br /><br /><strong>Become a Time Lord<br /></strong><a href="http://www.bbtestassistant.com/">BB test Assistant</a><br /><a target="new" href="http://cobrowse.mozdev.org/">CoBrowse </a>(a Mozilla Add-on)<br /><a target="new" href="http://windows.microsoft.com/en-NZ/windows7/products/features/system-restore">Windows XP mode - System Restore</a><br /><a target="new" href="http://msdn.microsoft.com/en-us/magazine/cc793968.aspx">Virtual Machines with Undo Disks</a><br /><a target="new" href="http://msdn.microsoft.com/en-us/library/ms189621.aspx">Database Backups</a><br /><a target="new" href="http://justgestures.com/">Gestures</a><br /><a target="new" href="http://www.fiddler2.com/SlickRun/">Slick Run </a><br /><a target="new" href="http://www.fitnesse.org/">FitNesse</a><br /><a target="new" href="http://www.greenpeppersoftware.com/confluence/display/GPW/Home/">Green Pepper</a><br /><a href="http://cukes.info/">Cucumber</a><br /><br /><strong>Cheat Shamelessly!</strong><br /><a target="new" href="http://www.seleniumhq.org/">Selenium</a><br /><a target="new" href="http://watir.com/">Web Application Testing in Ruby </a>(WatiR)<br /><a target="new" href="http://watin.org/">Web Application Testing in .NET </a>(WatiN)<br /><a target="new" href="http://getfirebug.com/">Firebug</a><br /><a target="new" href="http://www.scootersoftware.com/index.php">Beyond Compare </a><br /><a target="new" href="http://www.satisfice.com/tools.shtml">LogWatch</a><br /><a target="new" href="http://testing.gershon.info/reporter/">Rapid Reporter</a><br />Elisabeth Hendrickson's <a href="http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf">Test heuristics Cheat sheet </a><br /><br /><strong>Propaganda and the truth</strong><br /><a target="new" href="http://www.quest.com/toad-for-data-analysts/">Toad For Excel </a><br /><a target="new" href="http://en.wikipedia.org/wiki/Media_manipulation">Media Manipulation</a><br /><a target="new" href="http://www.tableausoftware.com/">Tableau Data Visualisation</a><br /><a target="new" href="http://www.splunk.com/">Splunk</a><br /><a target="new" href="http://www.hyperic.com/">Hyperic</a><br /><a target="new" href="http://www.nimsoft.com/">Nimsoft </a>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-64956182525823662642010-02-10T13:48:00.018+13:002010-02-12T13:57:02.582+13:00Reporting on VSTS and external perfmon dataGiven an ideal world, after any test run I would have just one centralised store of data the held all the performance related information. So, while the VSTS Load test data collation works really well,there are situations where you can’t avoid reporting across multiple data-sources.<br /><br />For instance, when testing locked down servers from outside the firewall (with no option to tunnel through), the only real option is to log the counters on the server-side and get these sent back for processing and correlation with the response stats(Grant H shows you <a href="http://blogs.msdn.com/granth/archive/2008/09/23/relogging-perfmon-binary-log-files-to-sql.aspx">here</a> how to re-format a binary log file into another format e.g. SQL via the ReLog tool, which is really handy for this kind of thing ). Another situation could be correlating a test run log file with some type of information (text etc) that isn’t suited to placing in a custom performance counter.<br /><br />For reporting across multiple datasources I have been extremely impressed by Tableau – a data visualisation tool, but I’m sure that this could be done with other charting software too.<br /><br />The key issues are –<br /><br /><br /><ul><li>creating a Union select statement that draws the results together across the two databases </li></ul>and<br /><br /><ul><li>ensuring that the timestamps play nicely together.</li></ul><p><br />I had issues with an invalid (line return or somesuch) character getting appended to the perfmon date time (hence the “LEFT([CounterDateTime], LEN([CounterDateTime])-1” to trim that off) and then needed to convert the formatted result into a datetime datatype for consistency with the VSTS data.<br /><br />If you have the Desktop professional version of Tableau you can manage the custom SQL directly within the Tableau data connection – I only have the personal version, so the data from the following query was drawn into excel as an intermediate step.<br /><br /><br />To get the VSTS data into a suitable form, here is a query that’ll compile a good data-set for this sort of view, it takes some trawling through the VSTS Beta 2 database schema to link this up, so here is a something to get you going....<br /><br /><span style="color:#009900;">--Select the perfmon data from the SQL database [PDB]<br /><span style="color:#3333ff;">SELECT Convert(datetime, LEFT([CounterDateTime], LEN([CounterDateTime])-1) ) AS time_stamp, ObjectName, CounterName, CounterValue<br />FROM [PDB].[dbo].[CounterData] Inner join [PDB].[dbo].CounterDetails on [CounterDetails].CounterID = [CounterData].CounterID<br /><br />Union</span><br /><br />--Select the VSTS data from the load test database [2010ResultsDB]<br /></span><span style="color:#000099;"><span style="color:#3333ff;">Select pd.TimeStamp as time_stamp, wrm.RequestUri as ObjectName, 'Response Time' as CounterName, pd.ResponseTime as CounterValue from [2010ResultsDB].dbo.LoadTestPageDetail as pd<br />Inner Join [2010ResultsDB].dbo.WebLoadTestRequestMap as wrm on wrm.RequestId = pd.PageId<br />Where pd.LoadTestRunId =</span> <span style="color:#cc0000;">::your test run id here::</span><br /></span><span style="color:#3333ff;">order by time_stamp Asc<br /></span><br />Here you can see the response time stats and (from VSTS) against the server CPU from perfmon.<br /><script type="text/javascript" src="http://public.tableausoftware.com/javascripts/api/viz_v1.js"></script><br /><object style="DISPLAY: none" class="tableauViz" width="600" height="600"><param name="name" value="VSTSresponsetimeandperfmonCPU/Sheet1"><param name="toolbar" value="yes"></object></p><noscript></noscript><div style="PADDING-BOTTOM: 0px; MARGIN-TOP: -6px; PADDING-LEFT: 0px; WIDTH: 675px; PADDING-RIGHT: 10px; FONT: 8pt verdana, helvetica, arial, sans-serif; HEIGHT: 22px; COLOR: black; PADDING-TOP: 0px"><div style="PADDING-LEFT: 559px"><a href="http://www.tableausoftware.com/public?ref=http://public.tableausoftware.com/views/VSTSresponsetimeandperfmonCPU/Sheet1" target="_blank">Powered by Tableau</a></div></div><br /><br /><br />If you have a lot of data it can really pay to create the suggested Tableau data extract which will speed up the rate at which the display refreshes when you dig around the results with the visualisations.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-80752169080083980032008-12-23T10:34:00.003+13:002008-12-23T10:41:32.891+13:00A new virtual horizon<p><a href="http://blogs.msdn.com/james_whittaker">James Whittaker</a> points to virtualisation one of the key things that will emerge in the future of testing and in my mind the Hyper-V revolution is definitely shaping up. At work here, we're already planning on ways to cope with private and shared VM "whole deployment" test environments as we iterate through the development cycles. Perhaps this is old-ish news to teams in product-focussed businesses. For Intergen as an IT service provider working in this space, with our small (days to weeks) test windows and many concurrent projects, this is huge.</p><p><br />I am also about to start execution of performance tests against a P to V (production to virtualised) environment - I remember talking to a client 18 months ago about how performance testing against a Virtualised environment should be treated as purely indicative, as the capability of VM's under load just didn't really approach that of real servers. Perhaps this is still true under stress, but the difference here is that we're talking about achieving acceptable "production-ready" response, and throughput, with only the CPU utilisation explicitly excluded from the set of acceptance criteria we'd use against the base metal. Funny how quickly things change. In any case, no doubt we'll learn something new.</p><p><br />I have a weather eye on the emergence of cloud computing Windows Azure / Amazon's EC3. Presuming that the application scales at all, I predict that these services will make it possible to scale out an app even if it is inefficient, just by buying more capacity. While you could worry that it threatens the 'Performance testing' discipline in some ways, I predict that it will make the justification for measurement and tuning more direct.</p><p><br />In the old days if it your app works acceptably well on the machines you already had (with some headspace) that was all you cared about. With the cloud, however, there is no fixed asset, so any performance improvements will save you operational cost pretty much immediately, and there will probably need to be a more regular assessment of your needs, especially if you have variability in your weekly / monthly / yearly demand.</p><p><br />In any case, this whole area is going to be new frontier, with a myriad of concerns not only technically but morally as well (e.g. legal jurisdiction, security/privacy). Exciting!</p><p>Feel free to share your thoughts on how VM technology is going to affect the tester.</p>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-82514188152389406372008-12-03T08:26:00.006+13:002008-12-03T08:44:08.773+13:00User Modelling and Test Automation<div>Not many people have talked about the "other side" of test modelling - what I call the user model, or the human proxy :-) And it's taken me some time to fall on this pattern to separate the input from the app model (decouple it) in a way that I'm happy with.<br /><br />My Test projects typically have 3 main pieces:<br /><br /></div><ul><br /><li>A test fixture/s .. the test/s themselves - uses both the page and user model </li><br /><li>An app model .. whatever test framework drives the app (i.e. WatiN /UIAutomation etc...) </li><br /><li>and a user model </li></ul><div><br />The user model is the mechanism I use to deliver data to the AUT. Think of the typical user view of the app - as a User of the app I have a username, password, email and dob etc. The user might need a shopping list which in itself might have many items, or only a few. They may need credit card with specific properites - Good, Expired, noAvailableCredit, weirdCardtype etc<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOk4_EtV1d-JKtlqtCfGngHhKL07YCVWGNPNKolLOCpOOfLK03faImiZP3DHKzp4I56Tgs2RpavawqsM7AGy4HSTIVCeIJHk3HOdJRKPu8eGeyJQLSYSsKGJ-s0RcLLKkjSqmvRg/s1600-h/AT-+2-12-2008+12.33.28+p.m..png"><img id="BLOGGER_PHOTO_ID_5275278252381202946" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 221px; CURSOR: hand; HEIGHT: 320px" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOk4_EtV1d-JKtlqtCfGngHhKL07YCVWGNPNKolLOCpOOfLK03faImiZP3DHKzp4I56Tgs2RpavawqsM7AGy4HSTIVCeIJHk3HOdJRKPu8eGeyJQLSYSsKGJ-s0RcLLKkjSqmvRg/s320/AT-+2-12-2008+12.33.28+p.m..png" border="0" /></a>So for my example app here, I have simple property bag classes of type User, PurchaseItem and CreditCard - with a User instance having a ShoppingList (made up of PurchaseItems) and a CreditCard. The nature of this model will depend on what your app does - a zoo management app would probably need animal, cage and refreshment-kiosk objects. </div><div> </div><div>Broadly speaking I end up with model object per form in the UI, although things like registering for an account and logging in will both use the "User" object.<br /><br />When I initialise a test I intantiate pieces of the user model and assign them values. Mostly the defaults will be fine, in other situations I construct specific data required for the test. By having mandatory data defaults built into the model (or even better dynamically retrieved from a known DB state and set in the fixture setup) - I'll keep clear of the "setup is good" vs "setup is bad" debate here...<br /><br />It means that the intent of THIS TEST is more obvious, as only the data that really matters for the test gets set here. I quite often find I use random generators where uniqueness is required. I have a set of helper classes in the user model to do this kind of thing with guids text dates etc.<br /><br />Once the user model is initialised I can start calling the page model and passing the user model components in as parameters.<br /><br />Here is an example....in this case the existingUser object, their default shopping list and the expiredMasterCard would be set in the setup fixture.<br /><br /></div><div><pre style="BORDER-RIGHT: #999999 1px dashed; PADDING-RIGHT: 5px; BORDER-TOP: #999999 1px dashed; PADDING-LEFT: 5px; FONT-SIZE: 12px; PADDING-BOTTOM: 5px; OVERFLOW: auto; BORDER-LEFT: #999999 1px dashed; WIDTH: 100%; COLOR: #000000; LINE-HEIGHT: 14px; PADDING-TOP: 5px; BORDER-BOTTOM: #999999 1px dashed; FONT-FAMILY: Andale Mono, Lucida Console, Monaco, fixed, monospace; BACKGROUND-COLOR: #eee"><code>[Test, Description("Purchase fails when attempting to use an expired CreditCard ")]<br /><br /> public void CreateNewAccountVerifyDetails()<br /><br /> {<br /><br /> //Setup Test Context<br /><br /> user.CreditCard = expiredMasterCard <br /><br /> using (HomePage homePage = new HomePage(ConfigurationHandler.PreProdUrl))<br /><br /> {<br /><br /> LoginPage loginPage = homepage.GoToLogin()<br /><br /> ProfilePage profilePage = loginPage.Login(user)<br /><br /> StorePage storePage = profilePage.GoToStore()<br /><br /> CheckOutPage checkOutPage = storePage(user.shoppinglist)<br /><br /> PaymentPage paymentPage = checkOutPage(user.CreditCard)<br /><br /> <br /><br /> Assert.IsTrue(enterPaymentDetails.ExpiredCardMessage())<br /><br /> <br /><br /> }<br /><br /></code></pre></div><div><br /><br /><br />This way the test code becomes pretty generic for the "login, select items and pay" flow. Only the expected result would change for different card info.Maybe you could even parameterise this further with something like the MBUnit rowtest pattern, one per card-type.<br /><br />Anyway this seems to work for me at the moment, but I'm sure there are many more ways to skin this cat. Let me know what works for you. </div>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-75483800573399001372008-11-05T13:49:00.003+13:002008-11-05T13:56:47.169+13:00VS2010 Testing Features - Part 1 CamanoSo two of our "strategy and innovation" team represented Intergen up at the Microsoft Professional Developers Conference in Los Angeles last week. They brought back a hard drive packed with goodies including the new Windows 7 OS and the latest CTP for VisualStudio 2010. I was super exited to get the Virtual Machine running today and play with the new testing features in VS2010 - there are several massive steps forward for QA pros (...and developers that care).<br /><br />We're a Microsoft shop. I've been using the MS VS tools since 2005 for web / load test projects and using the unit test types for hooking into WatiN and other frameworks.What has been lacking has been support for planning tests and managing the results. Camano is the intended solution and it's a standalone app that runs against a TFS repository, with Test artifacts all managed as workitems.<br /><br />The first step (and most important) is to get those document-based test plans/ test cases into a centralised tool that allows:<br /><ul><li>progress tracking</li><li>run reporting </li><li>bug / issue administration</li><li>It will also allow requirements and user story tracebility, if you go as far as managing these in TFS - which you should. (i.e. I can see you need not .. but you still should)</li></ul><br />By recording a manual test you can generate a number of artifacts including a video (to assist with communicating the activities that uncover bugs) and an "automation strip" for playback and also to act as a basis of generating a true automated test. .<br /><br />Playing with the features to "automate a manual test" reveals some of the more alpha quality areas, it just doesn't seem baked, although having said that the 1 GB restriction on my VM was not helping things (grindingly sloooow). I haven't quite grasped the Camano mechanism for running an automated test - perhaps that is just not there yet.<br /><br />To be honest though, these particular features are icing, and I'll live through some pain in that area to get the managment stuff I need. I also understand that the CTP was cut in July - so they are 3 or 4 months on from this at Redmond.<br /><br />Where things aren't quite there in Camano it does appear that TFS itself is further along. For example the reporting features and linking a test case with an automated test works in TFS but not in Camano. The only real downside as I see it is the need for a Team Foundation Server and the associated maintenance / cost overhead. I could see that being a battle for test teams that would otherwise win big out of going in this direction.<br /><br />While there are some complexities in learning how this works, it offers a powerful features. There are also some promises of "test environment" VM management in the Team Lab SKU.<br />Comparing to the Rational / Mercury players isn't easy but for the price point (ESPECIALLY if your org uses VS / TFS already) its likely to come out swinging...Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com1tag:blogger.com,1999:blog-12352206.post-4749728875815810822008-10-09T11:01:00.005+13:002008-10-09T11:09:25.884+13:00Learning and Understanding<p>I've just read <a href="http://media.pragprog.com/titles/ahptl/chap2.pdf">this sample chapter</a> from Andy Hunt's new book:<em> Pragmatic Thinking and Learning: Refactor Your "Wetware"</em>, that deals with the Dreyfus model of skill acquisition. While this kind of introspection is probably more the domain of <a href="http://blogs.msdn.com/micahel/default.aspx">Michael (the Braidy Tester) Hunter's blog</a>, I found it fascinating.<br />In particular it explained two things for me: </p><ol><li>That Agile Projects really do need those really skilled practioners driving them in order to succeed. These are the Jedi Masters that just "feel the force" rather than "read the manual". </li><li>That a lot of the anxiety I feel when learning something new comes from that fact that I don't yet have that fully-formed picture of the conceptual framework. I can almost feel that discomfort lessen each time I get one of those "a-ha" moments. Which interestingly occur not when something I'm trying works, but rather, when I understand why all the previous attempts didn't. </li></ol><p>I feel that this second point is important for testers to tune into - the better your internal model of the business/ techological processes, the less likely you are to miss important bugs in business/technological logic.<br /><br />I'm sure others will relate to this idea, let me know if you "feel it" too. Anyway I can't wait to digest the rest of the book.</p>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-58905548865309850502008-10-07T15:42:00.006+13:002008-12-23T10:41:52.105+13:00Visual Studio Web Tests - Random Access is not quite RandomI've just been grappling with an issue in VS2008 Tester Edition that was skewing the load profile I'm after for my latest performance test assignment.<br /><br />In a WebTest (either coded or UI-friendly) if you set "Random" access on Data Source and set the Number of Test Runs to a "Fixed Run count", the test runner will randomly select from all but the last row of data.<br />It works fine for 1 row, but with 2 or more rows, it will always ignore the last row. There is an n-1 bug in there somewhere....I've reported it to MS via the forums.<br /><br />I've checked and VS2005 seems to have the same issue, and it affects at least csv and xml data source types.<br /><br />The workaround is easy, as all you need is a padding row, but if you're wondering why the maths is off when using random access, this might be why.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-32946567195825975312008-08-29T15:55:00.004+12:002008-09-10T11:56:50.312+12:00Pairwise Data-Driven Automation - Post ScriptA caveat at this point: James Bach and Patrick Schroeder are of the opinion that pairwise testing should be approached with eyes open, and understanding the deficiencies of the approach, suggesting that random testing may be just as good. I've read their <a href= "http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf">excellent article</a> and I think I grok what they're saying. <br /><br />They propose that the effectiveness of pairwise testing depends on 7 factors:<br />1) The actual interdependencies among variables in the product under test.<br />2) The probability of any given combination of variables occurring in the field.<br />3) The severity of any given problem that may be triggered by a particular combination of<br />variables.<br />4) The particular variables you decide to combine.<br />5) The particular values of each variable you decide to test with.<br />6) The combinations of values you actually test.<br />7) Your ability to detect a problem if it occurs.<br /><br />The features of PICT address some of these points - not on it's own, but alongside robust analysis of the requirements and correctly implementing a model that follows them. The advantage it gives us in automation is that the model is external to the test (and therefore maintainable) and it generates the dataset rather than requiring a static source. <br /> <br />Anyway I've not seen anything else like this approach - I'd be interested to hear thoughts about whether you think it's useful.<br /><br />Future thoughts - I could dynamically generate some wrapper code to refer to the model elements, this would give me build errors if the expected paramenters weren't present in the source, and do the tilda stipping or any other input polishing outside the test.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-84202317381678711102008-07-22T12:15:00.008+12:002008-07-22T12:35:20.654+12:00Pairwise Data-Driven Automation - Part 3So at what we hope to achieve in this post is<br />1. Get a WatiN test going under the Visual Studio test framework<br />2. Hook the appropriate PICT output up as a data source for the test<br />3. Run the the test and see many inputs and many results running and passing. <br /><br /><br />1. First specify a data source from the Pict output file. You can also use the app.config to hook this up if you wish.<br /> i.e. <br /><br /> <!-- code formatted by http://manoli.net/csharpformat/ --><br /> <pre class="csharpcode"><br /> [TestClass]<br /> <span class="kwrd">public</span> <span class="kwrd">class</span> UnitTest1<br /> {<br /> <br /> [DataSource(<span class="str">"System.Data.OleDb"</span>, <br /> <span class="str">"Provider=Microsoft.Jet.OLEDB.4.0;</span><br /> <span class="str">Data Source=.;Extended Properties='text;</span> <br /> <span class="str">FMT=TabDelimited;HDR=YES'", </span><br /> <span class="str">"frombat#txt", </span><br /> DataAccessMethod.Sequential), <br /> TestMethod]<br /><br /> }<br /></pre><br /><br /> One gotcha here, you'll need to include the model input file and schema.ini (for tab delimted text files) in the test deployment settings <br /> ( Test > Edit Test Run Configurations > Deployment )<br /><br />2. I'll assume you have some familiarity - with WatiN, so build a test method that hits our login page and wire in the parameters from the Pict output.<br /><br />For our purposes we'll want to supply a bad username / password and verify that the right error message is returned.<br />Something like this...another gotcha - the tildes need to be stripped (I've left that in....)<br /><br /><!-- code formatted by http://manoli.net/csharpformat/ --><br /><pre class="csharpcode"><br /><span class="kwrd">public</span> <span class="kwrd">void</span> MainSiteIsUp()<br /> {<br /> <span class="kwrd">using</span> (IE ie = <span class="kwrd">new</span> IE(<span class="str">"http://www.actionthis.com"</span>))<br /> {<br /> <span class="rem">// Load variables from the file</span><br /> <br /> <span class="kwrd">string</span> email = context.DataRow[<span class="str">"EMAIL"</span>].ToString();<br /> <span class="rem">//strip the leading tilde if there is one...</span><br /> <span class="kwrd">if</span> (email.StartsWith(<span class="str">"~"</span>)) email = email.Remove(0, 1);<br /><br /> <span class="kwrd">string</span> password <br /> = context.DataRow[<span class="str">"PASSWORD"</span>].ToString();<br /> <span class="kwrd">string</span> message <br /> = context.DataRow[<span class="str">"$RESULT"</span>].ToString();<br /><br /> LoginPage.Login(ie, email, password);<br /> Assert.IsTrue(ie.Text.Contains(message));<br /> }</pre><br /><br />3. Select and run the test. It should build, cycle through the 7 combinations and check for the various error messages - all green!<br /><br />Yay - big pat on the back!Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com1tag:blogger.com,1999:blog-12352206.post-25031046493538362372008-06-24T13:17:00.005+12:002008-06-26T12:59:49.144+12:00Pairwise Data-Driven Automation - Part 2In this post I'm going to use the PICT model to define the "unsuccessful login" test cases for our application.<br /><br />I'll freely admit that the example I use here is probably not the strongest application of PICT, being relatively straightforward in the number of variants and potential cases it can output (i.e. I can keep all these rules in my head an construct a small number of tests to cover all of the variants anyway). Where it really comes into it's own is when there are more complex interactions and constraints that blow out the number of possible test combinations. I started looking at using it for Credit Card validation rules within the app, where there are a lot more rules regarding the minimum data requirements and the validation rules. But that's all internal logic, and the login screen anyone can see - so lets play with that.<br /><br />Let's start by defining the rules for the login screen.<br /><blockquote>To login you need a valid email address, a password and an account in the system with those credentials. The system gives one of three error messages different error messages </blockquote> ...depending on the rules we'll specify in the model file.<br /><br />Read up about the way that pict works in the user guide (comes with the <a href="http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi">download</a>) or the <a href="http://msdn.microsoft.com/en-us/library/cc150619.aspx">article i referred to last post</a>. Basically the above "spec" translates into a ModelFile like this, with 2 input parameters and an output parameter. I have taken a minimal set of invalid email address data to illustrate the point (i.e. there are others I'd include here if I were being throrough).<br /><br />#<br /># Login To ActionThis<br />#<br /><br />EMAIL: nodomain,,@domain.only, onepart@domain,correct@format.em.ail<br />PASSWORD: password,<br />$RESULT: The Email Address entered does not appear to be valid., Email Address and Password may not be blank., Login failed. Please check your username and password and try again.<br /><br /># Blank Email or Password rule<br />IF [EMAIL] = "" OR [PASSWORD] = "" <br />THEN [$RESULT] = "Email Address and Password may not be blank.";<br /><br /># Invalid Email address rule<br />IF [EMAIL] IN {"nodomain", "@domain.only", "onepart@domain" }<br />THEN [$RESULT] = "The Email Address entered does not appear to be valid.";<br /><br /># Valid email / password but no account rule<br />IF [EMAIL] = "correct@format.em.ail" AND [PASSWORD] = "password" <br />THEN [$RESULT] = "Login failed. Please check your username and password and try again.";<br /><br />Assuming you have pict installed, you can run pict directly from the commandline as : pict.exe ModelFile.txt > Outputfile.txt (try it and see). The output file is a tab delimted text file, which I'm intending to feed into the automation. <br /><br />So lets get a new C# test project happening in VS2008. I'll plumb the PICT in as a pre-build event for the project, so that any updates in the Modelfile are pulled through for the test run directly as and when that happens. <br /><br />One gotcha here is that running the pict.exe anywhere under the local solution directory gives me 9009 errors. Not sure why, possibly a PATH conflict? I was forced to running the pict.exe from it's native location (i.e. in C:\Program Files\PICT\) but referencing the files in the project folder. anyway, the pre-build event command line looks like :<br /><br />cd \"Program Files"\PICT\<br />pict.exe "$(ProjectDir)ModelFile.txt" > "$(ProjectDir)pictoutput.txt"<br /><br />OK so if your input file exists in the project dir, that should actually build and output a tab file, pictoutput.txt.<br /><br />Next up we'll wire it up to WatiN via a unit test in VisualStudio....Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-18148930018680648242008-06-20T17:21:00.005+12:002008-06-24T13:54:54.855+12:00Pairwise Data-Driven AutomationI've spent a bit of time in the last few days getting the WATIN Framework going with called PICT (the Pairwise Independent Combinatorial Testing tool), for pairwise test case generation.<br /><br />To date we've used hardcoded data values for most automated tests, and this leads to either embedded data all over the show (smelly!) and/or multiple calls to the execution logic handled within the test (inelegant)...and of course the coverage could be better.<br /><br />My first step towards a data-driven test used a "blunt force" attack on the data inputs (i.e an all-pairs approach, varying one input at a time) which generated a phenomenal number of test cases. As it was, our suite took quite some time to complete, but this approach just made it insane - anything that reduces execution time without compromising the new juicy coverage was going to be awesome. <br /><br />That's where PICT comes in. It's developed by a couple of engineers at Microsoft and is used internally by them top generate test cases "pairwise" in order to get the Input combination level most likely to find issues. You can read about the tool <a href="http://msdn.microsoft.com/en-us/library/cc150619.aspx">here</a> and download v3.3 from <a href="http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi">here</a>. Basically it offers a way to specify a set of outputs, their dependencies or constraints if any and also the expected results of those input(s). This gives us the ability to specify the input/output rules in one place (the PICT Input file) and get an output set of test cases optimised for coverage without redundancy.<br /><br />So where I'll get to over the next few posts is a rather trivial but nonetheless working sample of PICT / WATIN working together to give us data-driven automation, hopefully in a maintainable way. There were some issues that I've found I've had to work around on the way, so I'll let you know about those too!<br /><br />Next Up: I'll Dive in to the PICT Mapping Rules for our example.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-45532342048775822202007-05-23T20:36:00.000+12:002007-05-24T08:42:22.958+12:00Introducing ActionThis<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQTxJsbcGwtPtQRO6kKAAigaIQsMu-yCFztgGDY-3LxU1kW2JXyYPZF6qFjozuFfVWsv8-eb5IwphphIvd-wZWRyMcTQX85b6fxLKk9xcObL_2fiY_9tmPmnfqgGj2i9amHTPs4A/s1600-h/blacklogo.gif"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQTxJsbcGwtPtQRO6kKAAigaIQsMu-yCFztgGDY-3LxU1kW2JXyYPZF6qFjozuFfVWsv8-eb5IwphphIvd-wZWRyMcTQX85b6fxLKk9xcObL_2fiY_9tmPmnfqgGj2i9amHTPs4A/s320/blacklogo.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5067858565122264546" /></a><br /><br />Here's why I've been so quiet of late - work has been pretty absorbing. I'm part of the team working on an amazing new Web2.0-ey project called ActionThis.<br /><br />This is an application focussed on project execution and recovery - it's presenting exciting new Testing challenges partly because it represents a paradigm shift in it's approach - and partly because its the first truly agile software project I've been involved with.<br /><br />At <a href="http://www.intergen.co.nz">Intergen</a> we've now entered into a partnership arrangement with CompuWare. I'm using their TestPartner and QADirector in the day to day work on Action This. I expect to be presenting a <a href="http://www.intergen.co.nz/Hotlinks/Twilights/">Twilight session</a> shortly on this once we hit our Beta Release. I expect this to take the form of a Case study about the ActionThis QA process, automated testing and AJAX apps. <br /><br />It's an awesome thing to be a part of, and I can't wait to tell you more (when I'm allowed to), but you can sign up for more news as it happens at >> <a href="http://www.actionthis.com">www.actionthis.com</a>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-42306391100638601362007-02-02T14:02:00.000+13:002007-02-02T14:11:18.801+13:005 ThingsI got tagged by <a href=http://gabelog.net.nz/>Gabe</a> in the 5 things blog meme.<br /><br />I normally wouldn't do this kind of thing, but anyway, here goes:<br /><br />1. Tip-Top Pineapple Crush Icecream gives me hives. The world is now a safer place now they've stopped making it.<br /><br />2. I was in Rarotonga during Hurricane Sally (1 Jan 1987). The eye of the storm went over the top of us at halfway through. Apparantly this is relatively rare, but it gave us a chance to secure the section of roof which had lifted off. I was shit-scared.<br /><br />3. When I first came to Wellington I stayed in a flat full of Anarchists. Lots of mohawks, dogs and a dessicated pizza hanging from the living room ceiling. One thing that still makes me retch to this day was the fridge that the dog food was kept in. I'd guess it hadn't been cleaned in 5 years and their power was disconnected regularly. That relationship didn't last. Absolutely. Disgusting. <br /><br />4. After graduation, I got a job at Victoria University which involved (amongst other things) running and maintaining a Particle Accelerator for the Nuclear Physics Group. <br /><br />5. On my OE I spent months in Scotland without work. I could only afford to eat bread, jam and porridge, no heating and no fun. ( Actually looking back on it now it seems like a bizarre cultural immersion programme.) <br /><br />I agree with <a href=http://andrewpeters.wordpress.com>Mr Peters</a> - Everyone <strong>has</strong> already done this, so I won't spread the madness.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com0tag:blogger.com,1999:blog-12352206.post-4587915715708183572007-01-19T08:24:00.000+13:002007-01-19T09:21:28.164+13:00Watin / WatiNFixture - New Versions<a href="http://watin.sourceforge.net/">WatiN</a> 0.9.5 was relased 1/1/'07 with<br /><a href="http://sourceforge.net/project/showfiles.php?group_id=181273">WatiNFixture 0.2</a> following soon after at:<br /><br />I've now got a fairly good handle on a WatiN test pattern that works for me - basically a modified form of a pattern blogged by Richard Griffin. He's the guy that spooled up the WatiN Recorder tool to generate WatiN script from a browser session..<br /><br />My favourite new feature in WatiN 0.9.5 is the ability to filter out a collection of elements - this is particularly useful in repeater sections where an item of interest may be buried in a list with little to distinguish it from other items. I had previously been doing this by reg-ex matching within the HTML body and trying to strip out the relevant button ids etc.<br /><br />Much, much cleaner.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com1tag:blogger.com,1999:blog-12352206.post-1162439979946967132006-11-02T16:37:00.000+13:002006-11-02T17:14:22.366+13:00WatiN and WatiNFixtureI'm sure some of you will have heard of WatiR (Web application testing in Ruby). <br /><br />We'll I've recently discovered a .NET offshoot of the same (called WatiN), which suits: <br /><ul><br /><li>my limited programming ability by virtue of being C# scripted (although any .NET language will work). </li><br /><li>my .NET FitNesse environment </li><br /></ul><br />I've played around somewhat with using WatiN browser automation via VS2005 unit tests, and this works really well.<br />I also tried calling it via FitNesse through ActionFixtures and the like. It was cool, but a little heavy on reliance on the fixture programming.<br /><br />Anyway in trawling the FitNesse forums (<a href="http://tech.groups.yahoo.com/group/fitnesse/" target = "new"> http://tech.groups.yahoo.com/group/fitnesse/ </a> ), I stumbled across Jeff Parker's work on a dedicated FitNesse Fixture for WatiN automation, (un-surprisingly called WatiNFixture). This allows you to create an IE instance and control the actions and validation is a key-word manner. I hit him up for the code but it was getting some polishing and was likely to come out with WatiN 9.0 :-(<br /><br />WatiN release 9.0 was put out yesterday (<a href= "http://sourceforge.net/project/showfiles.php?group_id=167632" target = "new">here</a>) sans WatiNFixture. Things <u>are</u> really coming together, though - Frame / iFrame support and Dialog Box handling is all in this release. WatinFixture 0.1 also launched yesterday (<a href = "http://sourceforge.net/project/showfiles.php?group_id=181273" target = "new">here</a>) so it looks like they're collaborating rather than integrating at this stage.<br /><br />Anyway if you're into test / IE automation and Fit or FitNesse check them out.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com2tag:blogger.com,1999:blog-12352206.post-1160075276873053842006-10-06T07:49:00.000+13:002006-10-06T08:33:13.120+13:00FitNesse and Data ConversionI blogged quite some time ago about some data conversion testing I was doing. The customer has a freaky proto-database made up of hundereds of separate MS Excel worksheets and frequently running into the old Excel maximum row count limit. We're doing the sensible thing and getting it into a relational DB (in this case MSSQL 2005). Dont ask me how they managed up until now.<br /><br />I'm way overdue for an update on this, so here we go:<br /><br />I ended up working mostly with FitNesse and SQL stored procedures. <br /><br />I absolutely love FitNesse. For those of you who dont know it, FitNesse is based on <em>Fit</em> - a tool for writing and executing 'story tests' that both describe the desired behaviour of the system (requirements) and validate that the behaviour is achieved. <em>FitNesse</em> is a wiki based platform for writing and running these Fit tests. You can read more about it <a href ="http://www.fitnesse.org" target="new">here</a>. <br /><br />I had (the .NET 2.0 variant of) Fitnesse executing stored procedures retrieving data for given set of parameters, and checking that the result sets matched. The tests can involve many assertions, and they can be organised into Suites covering logical areas and suites of suites - covering pretty much the whole system.<br /><br />Given that there is absolutely masses of data (and conversion rules) I wanted to automate as much of these tests as possible, this meant I could keep working through the spec (chasing the dev) but every time I ran a test, I'd re-run ALL the old ones. Thing would break way back quite regularly. Once we got to feature complete, we just satrting doing bugfixes, and turning red tests into green ones.<br /><br />At this point (near the end I'm hoping) I have 2300 odd tests running with every migration pass. Of course I could have had some big arsed SQL scripts going and doing the same thing, but for me the beauty of the FitNesse tool is the following two points: <br /><br /><strong>Reporting </strong> - The outcome of the test runs is very obvious ( green / red colour coded to the test and assertion that has failed) <br /><br /><strong>Maintainability</strong> - The tests themselves (including the expected results) are easily editable. <br /><br />Oh ... and as they're "self documenting" pretty much circumvent writing test cases in Word. <br /><br />Im sold on it.<br /><br />Now I'm looking at getting some browser automation going with FitNesse. I'm looking at WatiN ( an I.E. automation framework like the Ruby <em>WATIR</em> framework, but for .NET ). More on this later.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1155858301263483252006-08-18T11:38:00.000+12:002006-08-18T11:45:13.080+12:00A disclaimerJerms is about to circulate a list of Intergen bloggers.<br /><br />It's probably appropriate that I say at this point that this blog is probably never going to be the central authority on Test Tools, Microsoft Test Tools or even Piers' dodgy hacked test tools.<br /><br />Even if I wanted it to be.<br /><br />I use it for collecting ideas and saving them where I can get at them later.<br /><br />...you may or may not be interested in going through my junk.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1139875686761250282006-02-14T13:01:00.001+13:002008-12-23T10:42:20.230+13:00The MSFT Web / Load Test GurusBefore I forget, here are the blogs for the Microsoft gurus on the VS2005 tester toolset<br /><a href="http://blogs.msdn.com/joshch/default.aspx">Josh Christie</a> - <a href="http://blogs.msdn.com/billbar/default.aspx">Bill Barnett</a> and <a href="http://blogs.msdn.com/edglas/default.aspx">Ed Glas</a>.<br /><br />Also:<br /><p><br /><a title="http://blogs.msdn.com/slumley" href="http://blogs.msdn.com/slumley">Sean Lumley</a> and </p><br /><p><a title="http://blogs.msdn.com/vstsloadtestblog" href="http://blogs.msdn.com/vstsloadtestblog">Team blog</a></p>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1137439170168530132006-01-17T08:09:00.000+13:002006-08-18T11:47:42.476+12:00MAF conversion - update 1So the MAF stuff:<br /><br />At the moment the system puts some data in to a database, and writes to an error log where it finds any issues. I haven't looked at the relationships in the target system or even the data quantity yet, but first things first....<br /><br />I scrounged this stored proc, which will count the number of rows in each table after the migration run has occured. At present these are stored in a temporary table and the results just written to the SQL Query analyser, but I'm looking to keep a record of these (outside the database itself - so as not to change the beast I'm checking). <br /><br />I stumbled across the bcp utility and the xp_cmdshell proc, which will let me write the results out to a text file of my choice - still feels a little inelegant, but I think it'll do for now.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1128298943762472012005-10-03T13:21:00.000+13:002005-12-09T15:19:30.320+13:00MAF - Data warsHere we go. I'm on MAF with the Featherston Street team. Prob'ly be there in the new year. <br /><br />Weird kind of testing this Data Migration - Building an application you hope will be run as few times as possible. I'm guessing that because there is no UI (and because Mr Andrew Peters is on-board) I'll be encouraged to implement a Fitnesse suite.<br /><br />Check out <a href="http://www.fitnesse.org" target - new>Fitnesse</a> here.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1126262234944324392005-09-09T22:36:00.000+12:002005-09-09T22:37:14.946+12:00VS2005 Web test custom validation rule/*********************************************************/<br /><br />using System;<br /><br />using System.Text;<br /><br />using System.ComponentModel;<br /><br />using System.Collections.Generic;<br /><br />using System.Collections.Specialized;<br /><br />using Microsoft.VisualStudio.QualityTools.WebTestFramework;<br /><br />using Microsoft.VisualStudio.QualityTools.WebTestFramework.Rules;<br /><br />/**********************************************************************************************/<br /><br />// Handle validation of response header values<br /><br />/**********************************************************************************************/<br /><br />namespace HeaderValidationRule<br /><br />{<br /><br />/**********************************************************************************************/<br /><br />// Inherits validation rule<br /><br />/**********************************************************************************************/<br /><br />public class ValidateHeader : ValidationRule<br /><br />{<br /><br />// Header name<br /><br />private string m_headerStringParameterName;<br /><br />public string HeaderStringParameterName<br /><br />{<br /><br />set<br /><br />{<br /><br />m_headerStringParameterName = value;<br /><br />}<br /><br />get<br /><br />{<br /><br />return m_headerStringParameterName;<br /><br />}<br /><br />}<br /><br />// Header value<br /><br />private string m_headerStringParameterExpectedValue;<br /><br />public string HeaderStringParameterExpectedValue<br /><br />{<br /><br />get<br /><br />{<br /><br />return m_headerStringParameterExpectedValue;<br /><br />}<br /><br />set<br /><br />{<br /><br />m_headerStringParameterExpectedValue = value;<br /><br />}<br /><br />}<br /><br />// Fail if not found?<br /><br />private bool m_isRequired = true;<br /><br />[DisplayName("Is Required")]<br /><br />public bool IsRequired<br /><br />{<br /><br />get<br /><br />{<br /><br />return m_isRequired;<br /><br />}<br /><br />set<br /><br />{<br /><br />m_isRequired = value;<br /><br />}<br /><br />}<br /><br />/**********************************************************************************************/<br /><br />// Add logic to handle header validation<br /><br />/**********************************************************************************************/<br /><br />public override void Validate(object sender, ValidationEventArgs e)<br /><br />{<br /><br />e.IsValid = false;<br /><br />e.Message = "Not found";<br /><br />for(int i = 0; i < e.Response.Headers.Keys.Count; ++i)<br /><br />{<br /><br />if (e.Response.Headers.GetKey(i) == HeaderStringParameterName)<br /><br />{<br /><br />string strVal = e.Response.Headers.Get(HeaderStringParameterName);<br /><br />if (strVal.Contains(HeaderStringParameterExpectedValue))<br /><br />{<br /><br />e.IsValid = true;<br /><br />e.Message = "Found";<br /><br />break;<br /><br />}<br /><br />}<br /><br />}<br /><br />}<br /><br />}<br /><br />}Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1126261733199912352005-09-09T22:26:00.000+12:002005-09-09T22:28:53.200+12:00Binding CSV data to a web test in VS2005(nicked from the VSTS Test Tools Forum)<br /><br />To add a csv file as a datasource, do the following:<br />1) Create a csv file that looks like the following:<br />username,password<br />user1,password1<br />user2,password2<br /><br />The first row is for column headers.<br /><br />2) Click the add datasource button for the webtest<br />3) Choose Microsoft Jet 4.0 OLE DB Provider as the OLE DB Provider<br />4) Click the Datalinks button<br />5) On the connection tab, enter the directory that the csv file is in for the "Select or enter a database name:" text box. Enter just the directory.<br />6) Click on the all tab.<br />7) Double Click Extended Properties<br />8) Enter text and hit OK<br />9) Click Ok for the Data Link Properties dialog<br />10) Click Ok for the connection properties dialog<br />11) Choose the csv file in the choose tables dialog. <br />12) Add the datasource to the field you want to bind to. The column headers in the csv file will be used for the field names.Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1126261215900629222005-09-09T22:17:00.000+12:002005-09-09T22:24:32.973+12:00The Perfect Bug<p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >(nicked from a VSTS Quality Tools blog)</span><span style=";font-family:Verdana;font-size:9;" ></span> <span style=";font-family:Verdana;font-size:9;" ><span style="font-weight: normal;"><span style=";font-family:Verdana;font-size:9;" >Whether we’re running automated or manual test cases, whenever we come across a failure it is often a good idea to log the issue in the bug database. In the Test Results window after a test run, you can use the list of failures to investigate a failure, rerun a test under the debugger, and finally associate the failure with a work item.</span><o:p></o:p></span> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >When you activate this feature, the product does a little bit of the work for you by opening a new bug form and filling in some of the fields.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >If you have access to a Team Foundation Server, I recommend trying this out.<span style=""> </span>First execute a test case that will fail, publish it, and then execute the “Create Work Item” menu item off of the context menu from the failed case.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:'Courier New';font-size:10;" >[<span style="color:teal;"><span style="color: rgb(0, 128, 0);">TestMethod</span></span>]<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:'Courier New';font-size:10;color:blue;" ><span style="color: rgb(0, 0, 255);">public</span></span><span style=";font-family:'Courier New';font-size:10;" ><span style="color: rgb(0, 0, 255);"> <span style="color:blue;">void</span></span> TestMethod1()<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:'Courier New';font-size:10;" >{<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:'Courier New';font-size:10;" ><span style=""> </span><span style="color:teal;"><span style="color: rgb(0, 128, 0);">Assert</span></span>.Fail(<span style="color:maroon;"><span style="color: rgb(165, 42, 42);">"Misc. bug in this code"</span></span>);<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:'Courier New';font-size:10;" >}</span><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >What you’ll see is a new bug form open with a bug title prefix (in my case “TestMethod1: “).<span style=""> </span>You’ll also see in the Comment and History section the error message text (in my case “Assert.Fail failed. Misc. bug in this code”).<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Note: Alternatively, if the bug you want to associate this failure with already exists, you can execute another menu item: Add to Work Item.<span style=""> </span>This helps you add failure information from this case to an existing bug.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >The rest, and arguably the <b style="">real</b> value, comes from you – we’ve just tried to automate some of the process that slows you down.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >So, what do you put into the bug?<span style=""> </span>Another way to ask this question is: what will the developer need to see in order to fix the bug in a very efficient manner?<span style=""> </span>What information will increase the likelihood of a bug fix?<span style=""> </span>How can I reduce the number of bugs that get resolved as Not Repro?<span style=""> </span>A process that helps you enter bugs that achieve all those things could be called <i style="">the Perfect Bug</i>.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><i style=""><span style=";font-family:Verdana;font-size:9;" >The Perfect Bug</span></i><span style=";font-family:Verdana;font-size:9;" > is a good thing to achieve.<span style=""> </span>Others will more quickly understand it.<span style=""> </span>Management will make quicker, but more informed and accurate decisions for the product.<span style=""> </span>The developer will be less likely to misinterpret the bug and provide the wrong fix.<span style=""> </span>You have supplied the developer with critical information that makes fixing the issue as quick and painless as is possible.<span style=""> </span>Everyone will spend less time on the bug (reading, comprehending, etc).<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >The <i>absolute</i> perfect bug is not usually attainable.<span style=""> </span>There is obviously a limit to the amount of time you should put into a bug.<span style=""> </span>There has to be a corresponding benefit to the time you put into it.<span style=""> </span>However, we can first focus on entering really good bugs and work our way up.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A high quality bug is easy to read.<span style=""> </span>It is concise, yet contains additional crucial information.<span style=""> </span>How can we communicate so concisely and clearly?<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Some of these items are specific rules to remember, but in general it can be wrapped up with a set of principles:<o:p></o:p></span></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Make it concise and easy to parse.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Considerably large data that would break the previous principle can be included elsewhere and simply referred to.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >It should be easy to find by another person (tester looking for duplicate, others looking for bug they once saw).<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >The bug entry should make the best and most accurate case for fixing.<o:p></o:p></span></div></li> </ul> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p> </o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A perfect bug has…<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><u><span style=";font-family:Verdana;font-size:9;" >An accurate and concise title<o:p></o:p></span></u></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A bug title should not be too generic.<span style=""> </span>The reader won’t understand what the bug really is.<span style=""> </span>Bad example: <i style=""><span style="color:green;"><span style="color: rgb(0, 100, 0);">App doesn’t work</span></span></i>.<span style=""> </span>What app?<span style=""> </span>How doesn’t it work?<span style=""> </span>What is specific about this scenario that causes it to happen?<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A bug title should not be too long.<span style=""> </span>The longer the title is, the more the reader has to concentrate; they may have to reread it several times.<span style=""> </span>Bad example: <i style=""><span style="color:green;">Leave defaults in a new test and run it.<span style=""> </span>Test outcome is "Failed".<span style=""> </span>Should it be "Error" or "Not Runnable" instead?</span></i><span style=""> </span>This really could be said in a lot fewer words.<span style=""> </span>Most of that title belongs in the Repro Steps section.<span style=""> </span>Try <i style=""><span style="color:green;">Defaults for new test yields a ‘Failed’ result</span></i>.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A bug title should include relevant error messages or crash address.<span style=""> </span>This makes it easier for others who are searching for duplicates. <span style=""> </span>Good example: <i style=""><span style="color:green;">FileNotFound Exception in ObjectStore.css line 47 when opening file with .xxx extension</span></i>.<span style=""> </span>If I get this error when testing and search on it, this bug title will immediately pop out at me.<span style=""> </span>Perfect!<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >A bug title should all words spelled correctly, especially error messages.<span style=""> </span>Take the time to make sure you spell words in your title correctly (do I hear an endorsement for built in spell check?).<span style=""> </span>People looking for duplicates will not find yours and this causes more work for everyone.<o:p></o:p></span></div></li> </ul> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><u><span style=";font-family:Verdana;font-size:9;" >Severity and Priority that are accurate<o:p></o:p></span></u></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Really think through the severity and priority you set for a bug.<span style=""> </span>Consider other bugs you have entered and compare this bug to those.<span style=""> </span>Know that developers (hopefully) use priority to set the order in which they fix bugs.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >If you enter a bug with what may seem like an unusual sev/pri or if the sev/pri are high, it would be very helpful to others if you explain your case for it.<span style=""> </span>Especially do this if you change pri/sev.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >As a product group, define what a bug means to be sev 1 and so forth.<span style=""> </span>The work item tracking solution has a feature to show a tool tip if you hover over the labels which can be used to reinforce the definitions.<span style=""> </span>Here are some typical definitions used by Microsoft for your benefit:<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Sev 1: Critical Failure.<span style=""> </span><b>Completely breaks product</b> or large set of features.<span style=""> </span><b>Unusable.</b><span style=""> </span>Significant risk or liability if release (security, legal).<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Sev 2: Major Impact / Functionality Broken.<span style=""> </span><b>Breaks major functionality</b> contributes to overall instability in this area, non-fatal assertions.<span style=""> </span>E.g., Statement Completion not active at all or memory leaks.<span style=""> </span>Regression from prior release.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Sev 3: Minor Impact / Functionality Impaired.<span style=""> </span><b>Breaks major functionality in a minor way or breaks minor functionality completely</b>.<span style=""> </span>E.g., missing item from list for statement completion.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Sev 4: Little / No user impact.<span style=""> </span><b>Can still use product / features</b>.<span style=""> </span>Minor functionality problems or UI blemishes or other issues that do not impact customers use or perception.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 0: “NOW” Bug.<span style=""> </span><b>Work stoppage, no work around</b>.<span style=""> </span>Blocking further progress in area or by group.<span style=""> </span><b>Fix Immediately!<span style=""> </span>24 hr turn around</b> expected!<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 1: <b>Showstopper</b>.<span style=""> </span>This is deeply impacting customers OR internal progress.<span style=""> </span><b>Worthy of a Service Pack or QFE</b>.<span style=""> </span>Fix Soon!<span style=""> </span>Also, required to fix for RTM.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 2: Important Bug.<span style=""> </span>Required fix for RTM.<span style=""> </span>Can be fixed any time before RTM.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 3: Something we would <b>like</b> <b>to fix</b> but <b>not required</b> <b>to fix</b> to ship the product.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 4: An unimportant bug or request.<span style=""> </span>A bug that will likely not be fixed.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >One manager at MS puts bug priority in terms that might resonate better with you for priority:<o:p></o:p></span></p> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 81pt;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 1: Will slip the product indefinitely to get this in.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 81pt;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 2: Will slip our date within limits to get this in. <span style=""> </span>Painful cut if not in.<o:p></o:p></span> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 81pt;"><span style=";font-family:Verdana;font-size:9;color:gray;" >Pri 3: Will not even think about slipping the product for any of these.<o:p></o:p></span></p></blockquote> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >If you are unsure about pri/sev, chat with others to level set your expectations.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >If you notice that management or others regularly resets your bugs’ sev/pri, ask them to discuss why they see it differently.</span><span style=";font-family:Verdana;font-size:9;" ></span></p></blockquote> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><u>Filled in fields (customize your bug form)<o:p></o:p></u></span></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >These are all examples of custom fields we use at Microsoft.<span style=""> </span>You can also add these to your own custom bug form.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >How Found<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >The idea is that the tester indicates how they found the bug, as in via what kind of testing.<span style=""> </span>Was it a <st1:place st="on"><st1:placename st="on">Test</st1:placename> <st1:placetype st="on">Pass</st1:placetype></st1:place>?<span style=""> </span>Automation?<span style=""> </span>Ad hoc Testing?<span style=""> </span>Customer Feedback?<span style=""> </span>Bug Bash?<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Your organization can use this field for metrics.<span style=""> </span>If you know you found 30% of your bugs via automation, it’s a compelling reason for your organization to invest more heavily in it.<span style=""> </span>If you find that bug bashes result in more bugs then you’ll know to plan more of those.<span style=""> </span>You get the idea.<o:p></o:p></span></p></blockquote> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Environment<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Can be very helpful with reproducing a bug.<span style=""> </span></span><span style=";font-family:Verdana;font-size:9;" lang="SV" >Enter OS, proc (32/64-bit), product flavor.<o:p></o:p></span></p></blockquote> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" lang="SV" >Blocking<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Is this a blocking bug?<span style=""> </span>You should definitely mark the bug as such and explain why in the description.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >A blocking bug usually is around one of these:<o:p></o:p></span></p></blockquote> <ol> <ol><li> <div class="MsoNormal" style="margin: 0in 0in 0pt 1in;"><span style=";font-family:Verdana;font-size:9;" >Precludes a build from being generally testable<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt 1in;"><span style=";font-family:Verdana;font-size:9;" >Prevents testing of other features in the area<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt 1in;"><span style=";font-family:Verdana;font-size:9;" >Precludes a build from being generally safe for dogfooding<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt 1in;"><span style=";font-family:Verdana;font-size:9;" >Breaks defined user scenario<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt 1in;"><span style=";font-family:Verdana;font-size:9;" >Degrades an feature area quality bar so that it is not meeting expectations</span></div></li></ol> </ol> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><u>Repro Steps<o:p></o:p></u></span></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >You can add a large, multi-line pane next to Comment and History to hold the repro steps.<span style=""> </span>Unlike Comment and History, this field can be updated at any time, whereas Comment and History can only be appended to.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >This is where you can make the biggest difference.<span style=""> </span>Repro steps tell the reader a LOT about the bug.<span style=""> </span>How easy to repro is this?<span style=""> </span>Is the customer likely to run into it?<span style=""> </span>Does it require the planets to align?<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Should include three sections: Repro Steps, Results, and Expected Results.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >If your actual steps to repro seem long, you will confuse the reader.<span style=""> </span>They will also think the likelihood of the bug impacting most customers is slight.<span style=""> </span>My suggestion is to include two repro steps.<span style=""> </span>Have one be how most customers will hit it.<span style=""> </span>The second one will be how to actually reproduce the bug from start to finish for a developer or another tester.<span style=""> </span>Blurring the line between these two often masks how likely a bug is to be encountered.<span style=""> </span>Make a good case for your bug to be fixed by describing the customer scenario separate from the straight steps to repro.<o:p></o:p></span></div></li> </ul> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><u>Extra info in the description<o:p></o:p></u></span></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Here is where you can put any other input you have.<span style=""> </span>Your expertise is highly valued.<span style=""> </span>Did you debug into the code a bit and find something relevant?<span style=""> </span>Put that here!<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Your bug may include a reference to the offending source.<span style=""> </span>Excellent example: <i style=""><span style="color:green;">Incorrect permission looked up - uses UPDATE_DATA permission instead of PUBLISH_DATA</span></i>.<span style=""> </span>The description could say, “Note: This is because <<i style="">path to file></i>\Service.cs:80 is requesting "UPDATE_DATA" rather than the publish permission.”<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >You could include a detailed description of the architectural flaws that led to the problem.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Do you have any root cause analysis?<span style=""> </span>Do you have any proposed fixes and comments about pros/cons of each fix?<o:p></o:p></span></p></blockquote> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Are there caveats to the bug that you think should be known?<span style=""> </span>Is the problem easily avoided or resolved?<span style=""> </span>Does the issue never go away?<span style=""> </span>What about after reopening the solution or restarting the IDE?<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >This is also where you can comment on your sev/pri rating. If this is a regression, here is where you should mark it.<span style=""> </span>A regression is a set of repro steps that used to work, either in a previous release or previous build that no longer works.<span style=""> </span>Management may be more likely to take a bug if quality has regressed.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Is this issue a crashing bug?<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >If the OS has crashed (blue screen), do you have a Kernel Dump or a Kernel Mode Debugger attached to a repro machine?<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >UI crashing bugs should have a call-stack and mini-dump.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >ICE (internal compiler error) bugs should have a preprocessed file.<span style=""> </span>If a CHK build is available does the crash repro on CHK, and are there any Asserts hit?<span style=""> </span>When in doubt, keep the debug session active and contact the owning dev.<o:p></o:p></span></p></blockquote> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >You could include a list of test cases to run to verify the fix.<span style=""> </span>Oooh, ahhhh.<o:p></o:p></span></div> </li><li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >Still very important… make all of the above readable.<span style=""> </span>Format it in such a way that one can read through the bug and quickly identify relevant information.<o:p></o:p></span></div></li> </ul> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" ><u>Attached files<o:p></o:p></u></span></p> <ul> <li> <div class="MsoNormal" style="margin: 0in 0in 0pt;"><span style=";font-family:Verdana;font-size:9;" >There are several different kinds of files that may be very relevant.<span style=""> </span>Perhaps the most helpful is a screenshot of UI that is bugged, especially in localization issues.<span style=""> </span>Often you can say a lot more with a picture than you can accurately convey in text.<span style=""> </span>Indicate that you’ve attached a picture so others know to look in the Files tab.<o:p></o:p></span></div></li> </ul> <blockquote style="margin-right: 0px;" dir="ltr"> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >The picture should be JPG or PNG format. <span style=""> </span>BMP and other formats are generally too big.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style=";font-family:Verdana;font-size:9;" >Crop the picture to what is needed to show.<o:p></o:p></span></p></blockquote> <span style=";font-family:Verdana;font-size:9;" >If you have a lot of code to include in a repro step, it might be best to simply attach a file and refer to it in the bug Repro Steps.</span><b> </b>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1124909772341726662005-08-25T06:25:00.001+12:002008-12-23T10:42:44.410+13:00VS2005 Load Testing<b></b>Ok, so the ACT testing didn't go so great. Basically the Testing window was 4 hours (from 3p.m. to 7p.m.) and there were some significant responsivity issues I saw during that period. Anecdotally we saw a huge improvement after changing (a seemingly un-related) registry setting regarding the Active Directory implementation, but who knows? (we didn't re-test...)<br /><br />I've been asked to look at the performance for another client and have spent quite some time with the VS2005 Beta 2 and getting a series of tests automated. It is a huge step up from the ACT tool although grappling with the most beta aspects is still a challenge.<br /><br />Things I really like:<br /><ul><li>The web test recorder works with HTTP 1.1, SSL and NTLM etc...</li><li>The automatic handling of VIEWSTATE (thankyou, thankyou, thankyou)</li><li>Out of the box extraction / validation rules and their extensibility (very powerful).</li><li>Working in an Intellisense IDE.</li><li>Visual representation of the requests, querystrings, form posts etc</li></ul>I'll just quickly list some of the areas I've had some pain - if there is anyone out there who needs more info about any of these, i'm happy to share....<br /><ol><li><strong>XML data binding</strong> - Nope not supported just yet as far as I can tell - use an SQL table if it's required.</li><li><strong>Web /Load Test renaming and moving</strong> - also works inconsistently. I get warnings and misnamed files quite regularly when doing these prety basic operations. I've learned to be careful about this and usually any problems are obvious immediately. I just kill the copy and re-try until the tool gets it right (which is does eventually).</li><li><strong>Load Test Results</strong> - Don't seem to be saved in the same way as other test type results are. When I end a test session and shutdown VS2005, the result view basically disappears. I suspect that there is something more I could do if we had the ability to publish the results to the Foundation Reporting service, but I'm not sure. The backend data can be pulled from the DB still though, but the front end presentation is so good. What I'm intending to do to work-around this is run the tests in VS2005 but have perf mon record the counters separately.</li><li><strong>Test Result DB connectivity</strong> - This required some in-depth research to get running. There is information out there around this (<a href="http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=55303">http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=55303</a>) but the conection string dialog is still flakey. I did have to re-install the tool to get back at the appropriate setting when I wanted to re-set it.</li><li><strong>Random crashes</strong> - these happen a lot with test creation and de-bugging -although I must admit that I haven't lost data as a result, and I haven't had to abort a load test run in this manner either....When running web tests it is common to get the system "freezing".</li><li><strong>The API is largely undocumented</strong> <strong>as yet</strong>- a pain if you want to do anything beyond what MSFT have provided for.<br /></li></ol><b></b>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.comtag:blogger.com,1999:blog-12352206.post-1116728901047088732005-05-22T14:00:00.001+12:002008-12-23T10:44:27.076+13:00Start at Intergen and perfomance testingThis is my first post since joining Intergen. A pretty scary first week just trying to get to grips with the what and how of the projects and company ... seems like a good bunch of people, though.<br /><br />I'm going to have to dust off the VBScript skills for the perfomance testing I'm doing with Damon. The ACT tool has some serious shortcomings in it's 'development environment' which will have the side effect of making me think HARD. This will be kind of good because it will drum in some understanding, but will probably take longer than the PM would like. In any case, it's nice to have some under the hood work to do and is precisely the kind of thing that MoE just didn't get near.<br /><br />A couple of things that DC mentioned that seemed important so I will note them here while I remember.<br /><ol><li>DC mentioned that the is a machine config setting that needs to be watched on the webserver under test. I think in the web server machine.config file: the enableViewStateMac="false"...</li><li>The Client machine needs to be quite good for even emulating 100 concurrent users. As the spec is calling for 2400 for extreme load this will be interesting to try on anything that isn't massive.</li><li>I'll need to ensure I get permissions on the target web and db servers so I have access to the performance metrics.<br /></li></ol><a href="https://www.se.auckland.ac.nz/uploads/researchProjects/ArgoMTE-1.pdf">Here</a> is an interesting white paper from AK uni, which have yet to digest... and an interesting idea for a <a href="http://http//dotnetjunkies.com/WebLog/tim.weaver/archive/2004/10/28/30265.aspx">converted development framework</a>.<br /><b><br /><br /></b>Pierzapinhttp://www.blogger.com/profile/12352092988194261301noreply@blogger.com