I am speaking at the first NZ ALM Conference next month in Wellington.
I will be doing 2 sessions.
Testing in an Agile World or Integrating BDD Frameworks with MSTest and Team Foundation Server
These sessions will be recorded, published and available to view online if you can’t make the conference. Leave a comment if you have a preference.
Continuing our recent BDD theme I thought that it would be useful to show an example of combining SpecFlow and coded UI tests for automated acceptance testing using the UI. SpecFlow is a popular open source Acceptance Test Driven Development and Behaviour Driven Development framework for .NET.
There are some posts out there about how to do this, but no step by step examples. I thought that I would fill in the gaps and also submit the code to the SpecFl0w guys.
An example of a SpecFlow scenario is
As you can see it is plain text with no testing framework ‘noise’ getting in the way.
With SpecFlow you can generate .feature files that can be executed by a variety of test runners. Features contain multiple scenarios which get turned into unit tests. Out of the box SpecFlow supports NUnit, MSTest and xUnit. We are going to add support for CodedUI.
I have downloaded the SpecFlow examples and we will be working with the BookStore sample.
I have created a new Visual Studio 2010 Test Project called BookShop.AcceptanceTests.CodedUI. I have copied the structure of the other samples and re-created the same folders.
Copy in the ReferenceBookList class and BookSteps class. These should work as is.
There is a great post here which gets us started. Follow these instructions to build a new generator that will decorate the tests for Coded UI. I have included this with the attached code. You will need to copy this dll (SpecFlow.CodedUI.Generator.dll) into your SpecFlow bin folder.
Looking at the Selenium sample we can see what sort of coded UI tests we need to create.
Firstly we need a test to perform the simple search. I launch a new coded UI and click into the search field, enter a title and then hit search. At this point generate the code. It is a good idea with coded UI to keep the methods really small. That way you can compose them in different ways.
I have called this method PerformSimpleSearch. We also need a way to validate that the list should exactly contain a certain book. Start the coded UI editor again and create as assertion on the html cell with the name of the book. I have called this TitleMatches. I also created an assertion on the rows property of the table. We can use this to make sure that there are no extra books in the list.
I created one more method in my coded UI test and that was to close the browser. Called CloseBrowser (funnily enough)
Right thats all the coded UI tests we need. Now we can wire these up to our SpecFlow project. If we copy the outline of the SearchSteps class from the selenium project we see:-
[When(@"I perform a simple search on '(.*)'")]
public void PerformSimpleSearch(string title)
{
}From here we can insert the calls to the Coded UI functions to validate behavior. So inside this method we can add
codedUI.PerformSimpleSearchParams.UISearchTermEditText = title;
codedUI.PerformSimpleSearch();This uses the feature of coded UI that enables us to specify parameters to methods using the Params properties. So we can leverage both the power of coded UI and SpecFlow together. The next method looks like this:-
[Then(@"the book list should exactly contain book '(.*)'")]
public void ThenTheBookListShouldExactlyContainBook(string title)
{
}again we can fill this in with
codedUI.HasExactBookCountExpectedValues.UIItemTableRowCount = 2; //one for header row
codedUI.HasExactBookCount();
codedUI.TitleMatchesExpectedValues.UIDomainDrivenDesignCellInnerText = title;
codedUI.TitleMatches();
codedUI.CloseBrowser();Great, so now we can run our first SpecFlow test. Navigating to the TestView window we can see
And running it shows us SpecFlows great readable output.
I have also updated the code to make the second test work. See the attached sample.
Code is here.
We left Part 6 with our registry changes to get impact analysis working, however we still needed to restart IIS manually in order to reset the test run.
I promised that I would investigate writing a custom Data Collector to do this for us, and here it is.
Here is a great MSDN article on How to: Create a Diagnostic Data Adapter. I followed this and added the code we needed to restart IIS when the test run starts.
ServiceController iisService = new ServiceController("W3SVC");
if (iisService.Status != ServiceControllerStatus.Stopped)
{
iisService.Stop();
iisService.WaitForStatus(ServiceControllerStatus.Stopped, TimeSpan.FromMilliseconds(15000));
}
iisService.Start();
iisService.WaitForStatus(ServiceControllerStatus.Running,TimeSpan.FromMilliseconds(15000));This is pretty self explanatory so I won’t go into the details.
Ok, now we have our adapter lets install it. Back to MSDN – How to: Install a Custom Diagnostic Data Adapter. We basically need to copy the adapter into the DataCollectors folder on both the test agent machine (the one running our website) and restart the test agent.
Once we do this we have another option in our TestSettings screen in MTM.
Make sure that this box is ticked. Now lets run our test again. When we hit run test in MTM the custom data collector should kick in and restart IIS on the server. So lets do that and check the event log on the server. Mine correctly shows two events (in the system log) – WWW Publishing Service stopping and starting.
Now we can look at the Application log and we should see some information from the Profiler. Here it is:-
The profiler has started. To double check I have opened the W3Wp.exe process with Process Explorer to check the environment variables. As we expected we have the COR_PROFILER and COR_ENABLE_PROFILING variables set correctly.
We now run our tests and check the results in MTM.
Here we can see our test impact results have been generated!! Double click this and we see:-
We have successfully captured TIA data! The final check is on the server. Go back and look at the application log again. We need to ensure that the profiler is being shutdown. Sure enough here we see:-
Success!! We now have well functioning TIA data with remote IIS.
Fitnesse is an Acceptance Testing Framework that uses a table driven approach. It isn’t really one of the cool kids anymore having lost that title to the likes of SpecFlow, StoryQ and other BDD based frameworks.
You can however use FitNesse for BDD style tests. Here is a video from uncle bob on the subject.
One of the main reasons that I still like FitNesse is that it is tester friendly. By this I mean that non-coding testers seem to take to it well.
I am going to assume that you are familiar with Team Build, FitNesse (and Slim) for the remainder of the post.
I am also going to assume that you know the basics of MTM.
Another feature that I wanted to try out as part of this was the Build-Deploy-Test workflow on physical environments. This has just been released by Microsoft (CAUTION: UNSUPPORTED)
I have taken from this excellent post from Jerry Bullard.
Also the workflow for Build-Deploy-Test was published here
Follow the steps in Darshans blog to get your environment setup. NB Remember to configure the Test Agent as process rather than a service. It will need to interact with the desktop to invoke FitNesse. Stop once you have the environment configured in MTM like this.
We will write an exe that wraps the REST call to Fitnesse. This exe will then transform the Fitnesse result using an XSLT file. The exe will be called from a Generic Test. This gives us VS/TFS integration in terms of deployment, running etc. All we then need to do is make sure the results are in a Summary Result Format.
I have used a Slim tutorial sample and have one test passing, one wrong and two ignored. This should give us a good sample of results.
There are a few utilities that you will need to get this solution working. I have attached all of them (including source) to the link at the end of the post. None of the code is exciting or interesting so I am not going to go through it here. The utilities do the following:-
GeneralFitTestRunner.exe – executes the FitNesse tests using the REST api. Arguments are
Usage example:- GeneralFitTestRunner.exe 127.0.0.1 8082 FrontPage test
ProcessStarter.exe – starts a process. Arguments are
Usage example :- ProcessStarter.exe java.exe –jar c:\fitnesse\fitnesse.jar –p 8082
Run.bat – This invokes the ProcessStarter.exe and passes in the arguments to start FitNesse. You may need to change the paths in here for your environment.
CopyWiki.bat – Uses xcopy to copy the FitNesse wiki from the droplocation of the CI Build onto the integration server. You may need to change the paths in here for your environment.
Stop.bat – Stops FitNesse.
All these scripts assume that FitNesse is installed on the c:\FitNesse directory on the integration server.
Create a folder called c:\Tools on your integration server and copy all of the utilities into it except run.bat and stop.bat. Here is the folder on my server.
Place run.bat and stop.bat into a directory that can be referenced via a UNC from both your development machine and the integration server.
We can now create our Generic Test. If you open the TestFitIntegration project and look at the FitTest generic test you will see:-
We are pointing to our GeneralFitTestRunner and passing the arguments to run the FiNesse test.
Note the Additional files to deploy section:-
This is the file that FitNesse will use to execute its tests. i.e.
We need to add this to (at least one) generic test to make sure that it gets deployed for us.
The Results settings indicate that we will generate a Summary Results file. Make sure that the name is Results.xml as shown.
My solution file contains the following FitNesse artefacts.
I am using this for deployment. I have customised my CI build to copy the FitNesse folder to my build drop folder. I am not going to step by step this. The xaml is attached so you can use the same one.
If you don’t do it this way you will need to find a way to make sure that your latest wiki content gets pushed to your integration server.
Now we need to start building our test plan. You should already have an environment setup so go ahead and create a new Test Plan and Suite. Mine looks like this:-
Opening the test shows us:-
If we navigate to the Associated Automation tab you can see that I have associated my generic test with this test case. Here is one way to do this.
Next we have to sort out the Test Settings and Environment. If you choose Properties of the plan
From here we can update our Test Environment. Choose the environment that you have created. Make sure that this is for Automated Runs
Click the Open link next to the Test Settings for Automated Runs.
Ensure that the Radio button is selected for Automated. Select the Role that you assigned the machine in your environment.
Choose any relevant Diagnostic collectors
Click on the Deployment menu underneath the Advanced option:-
I am using this to deploy the FitNesse folder (just the wiki) that has been copied to a UNC share from my CI Build – see above) This ensures that I am running the latest version of my wiki.
Next move to Scripts.
These are the run and stop.bat scripts that we copied to an available UNC share earlier.
The run.bat script invokes the ProcessStarter exe which runs java – jar etc. The reason for needing this is the way test settings work. The test will not be run until the script finishes. Obviously this is not good for us as FitNesse will stay open – and we need it to. So we need the script to fire off an exe and then exit. This will load FitNesse and run our test.
N.B. Starting FitNesse this way means that it ‘unpacks’ each time. To cope with that I have added a delay in the ProcessStarter. A better solution would be to read the output and see when its finished unpacking.
The stop.bat stops FitNesse.
We now need to setup our builds.
The flow that we are looking for is this:-
Create a new build definition for our CI Build. Use the RunIntegrationBuildProcessTemplate.xaml that I have provided. This should build the TestFitIntegration solution that I have provided. In your build definition you will need to specify a directory to copy the FitNesse wiki to. This should be a UNC accessible folder. This folder is referenced in the copywiki.bat script. Make sure that these folders match.
Create a new build definition and use the BuildDeployTest_Physical.xaml file.
My Integration build is configured like this:-
As you can see I have targeted a specific number of my CI Build (20110316.28.) This is just for testing. Unfortunately this xaml does not support just getting the latest build, but depending on your build numbering that would be a trivial change to make. The other option is to have the integration build run the CI build first. If I chose this option my config would look like this:-
We need to specify some properties for this build. Firstly choose whether you want your CI build to run first or target a specific build. These sections (1,2 and 3) are fairly self explanatory – if you need any more detail have a look at Darshans blog. In Section 4 – Deployment – we want to add our deployment script – copywiki.bat. See the screenshot above for an example. We give the Build Agent Name that we want the script to execute on, and the directory and name of the script. Make sure to set the Run Deployment Scripts parameter to True.
Section 5 is for testing. Set Run Tests to True. Test Environment name is the MTM environment that we setup in the beginning. The Test Run Parameters has a nice dialog to help us. Select the test plan and suite that we created in MTM. Make sure that you select the correct Test Settings.
Right that’s it!
A successful integration build looks like this:-
As you can see we have details of what build we have deployed from (our CI build) and the test run details. Clicking on the View Test Results link opens MTM with the following:-
Great! So we can see the actual FitNesse breakdown of results. Because this was executed by a Test Plan we get all the features of MTM e.g. it will show up in the Analyse Test Runs View.
Not to mention all the testing reporting. (But that’s another post) 
Hopefully this has shed a bit of light on two main techniques. Firstly how we can execute an external test runner like FitNesse, but also how we can use the (unsupported remember) Build Deploy Test xaml on physical environments.
Links to all code here.
BDD has been gaining traction for a long time and MSpec is one of my favourite frameworks.
Unfortunately there is no compatibility with MSTest for running within Visual Studio. This also means that you cant run it easily using Team Build and have your test results recorded as part of the build. Well now you can, as I am going to show you how to integrate MSpec and Team Build.
Firstly a disclaimer:-
And now some acknowledgements:-
This post will assume that you are comfortable customising Team Build etc. I will try and include help links where I can but I am not going to explain the whole step by step process.
You can either follow along, or just grab the attached files and go to the How To Configure section to get it working for you.
We have a nice head start because even though this was written for NUnit, it is easy to extend. There is an xslt file that transforms from the NUnit runner format to the trx format. I wrote another xslt that transforms from MSpec. The other code works pretty much untouched. Unfortunately the MSpec xml output is not as rich as either MSTest or NUnit and so lots of information cant be populated.
The one change that I had to make concerns the test run start date. This information is not supplied by the MSpec runner. You also cannot create a date value with xslt 1.0. The code in the NUnitTFS library expected a certain format. So I took this function
private static DateTime GetStartDate(XDocument trx)
{
var name = GetName(trx);
return DateTime.ParseExact(name.Substring(name.IndexOf(' ') + 1), "yyyy-MM-dd HH:mm:ss", null);
}and changed it to
private static DateTime GetStartDate(XDocument trx)
{
var name = GetName(trx);
try
{
return DateTime.ParseExact(name.Substring(name.IndexOf(' ') + 1), "yyyy-MM-dd HH:mm:ss", null);
}
catch (System.FormatException)
{
return DateTime.Now;
}}
This will return the current datetime for MSpec tests.
Everything else is a command line option. I will go into these when we look at the xaml file.
Build machine Changes
Obviously MSpec will need to be installed onto the build server. I copied all of the binarys into c:\Machine.Specifications-release. I also copied NUnitTFS into this folder.
Build Changes
Open the xaml attached to this blog and navigate down to the Try Run Tests section. There are 2 copies of the Run MSpec Tests shape in both branches of MSTest execution. This ensures that regardless of TestLists / TestSettings our MSpec tests will be run. The Run MSpec shape looks like this
The shapes break down like this. The FindMSpec Tests mimics the MSTest method of using a regex expression to locate the correct test assemblies.
A build definition parameter has been created to take input from the user on the correct regex. Again this is to mimic the MSTest behaviour.
A variable called MSpecAssemblies has been created in the build workflow to hold a List<string>. This is the names of the assemblies that match the regex.
The rest of the workflow (with the expanded Run MSpec) is here.
If Test Assemblies Found checks if any test assemblies match the Regex. If so we need to flatten our List<String> containing the assemblies to run. We need to pass them to the MSpec test runner.
This shape builds up a string containing the list of assemblies.
We are now ready to call the MSpec test runner. Our Invoke Process parameters look like this:-
Arguments
Nothing should be surprising here. By putting the results file into the build output directory (binaries) we are making sure that there shouldn’t be any interference across builds. The last parameter is our list of MSpec test assemblies.
Publish MSpec results
This is our call to NUnitTFS.exe to transform and publish our test results.
Arguments:-
These parameters should be pretty self explanatory. A word of warning here, you will need to specify a configuration on your build definition as below.
The last if shape should be pretty clear. If the tests do not pass then the build will update the status, so that the build will be marked as partially successful.
Whew, ok so lets now run it.
Copy MSpec onto your build machine.
Copy NUnitTFS onto your build machine (ensure that it is a modified version – see above.)
Update the NUnitTFS.exe.config urls with your TFS addresses, e.g. change http://tfs:8080/tfs/TestManagement/v1.0/TestResults.asmx to point to your tfs instance. There are 3 urls to update.
Create an new build definition. Make sure that it has a configuration set.
Complete the following properties in the build definition.
This is the path to the MSpec test runner on the build machine, the regex format to find MSpec test assemblies, the folder where the xslt resides, and finally the path to the NUnitTFS.exe.
Run the build.
Here is a screenshot of a finished build that contains a single MSTest and multiple MSpec test assemblies.
When you click the View Test Results link you see:-
These are our BDD tests! If we double click on a failed test we see the following:-
We can drill into the source just as with MSTest! Success!
It might be nice if all of the xaml was packaged up into a custom activity. I will leave this as an exercise for the future…
xaml - here
NUnitTFS - here
Spoiler – Link to final xaml is here
If your tests take a long time to run, you may wish to only run the tests that have been impacted by code changes checked in with the build (and of course run a full nightly build that executes all tests :) Unfortunately there is no out of the box feature to do this, so we need to edit our build xaml to do it. Lets get started:-
If you scroll down the workflow and find the “Get Impacted Tests, Index Sources and Publish Symbols” parallel shape you will see that the check for impacted tests happens after the run tests activity. This means that if we use the xaml out of the box all of the tests will run every time.
We need to reverse this order. If we can scope the Run Tests shape with the output of the Get Impacted Tests shape then we should be all good.
This just leaves one problem. At various points (maybe every night, maybe after each release) we need to gather the data for impact analyis by running all of the tests (to include any code changes since we last ran it) This should be configurable.
Right lets get started. Find the "If PerformTestImpactAnalysis" parallel shape as below:-
Drag this shape upwards and insert here:- 
Things should now look like this:-
By moving this shape we have ensured that the Impact Analysis will occur before any tests are run. We now need to somehow scope the tests to only those that have been impacted by changes in the build. Fortunately we are in luck. The "GetImpactedTests" shape (msdn ref) has an output parameter called (funnily enough) ImpactedTests. This is defined as List<Test>The "MSTest" shape (msdn ref) has an optional input parameter called TestNames. This is defined as IEnumerable<String>
All we need to do is enumerate through the list of tests returned by the GetImpactedTests shape and pass them as a list of strings to the TestNames parameter of the MSTest activity. The steps to do this are:-
We need to convert the TestList into a list of strings for MSTest. To do this we will use a ForEach shape to enumerate the List and an AddToCollection shape to build the list of strings. We can then pass this list to MSTest.
We need to pass this list of tests into the Run Test shape. To do this find the “Try Run Tests” Try Catch shape below our current position. Impact analysis only works when MSTest is run with a test settings file so we only need to update the Run Test activity in the middle. i.e. the middle red circle below.
Set the TestNames property of the “Run MSTest” shapes to our TestNames variable.
This just leaves us with two problems. Firstly how do we get all of the tests to run for the first build? Otherwise we will never have any impacted tests. The second problem is how do we ‘reset’ the impact data to include new code. We may want to do this nightly or every release/sprint etc. Fortunately we can solve both of these problems with the same solution. We need a way to override the scoping. If the TestNames list is empty then MSTest will run all of our tests. This solves our problem but introduces an edge case. What happens when there are no tests impacted by our code changes. We don't want any tests to run in that case.
We can solve this by using a custom process parameter. Follow the guidance here to add this to the Process tab of the definition editor. I called my parameter Is Base Line Run, and placed it in the advanced section. The parameter is a Boolean.
We can now add an If shape like this:-
The Condition property on the If shape is:-
TestNames.Count = 0 And BaseLineRun = False
In the “Then” portion of the shape we add an Assign shape and set DisableTests = True.
This will mean that if there are no tests in the TestNames List and it isn’t a base line run then no tests will run (because nothing was impacted) However if the base line run parameter is set to True then all of the tests will run.
Right, time to queue a new build. Remember to set the Is Base Line Run parameter to True for the first build. This should run all of your tests. Here is the build output for my first build.
This ran all 7 tests in my solution.
Now lets change some code. Here is the Test Impact View in Visual Studio showing that my changes have impacted two tests. I now checkin my changes.
Now I queue another build, leaving the Is Base Line Run at False (the default.) The build output now shows:-
Success!! We could setup a nightly build with the Is Base Line Run parameter set to True and have out impact data being updated every night.
Link to final xaml is here
