Spoiler – Link to final xaml is here
If your tests take a long time to run, you may wish to only run the tests that have been impacted by code changes checked in with the build (and of course run a full nightly build that executes all tests :) Unfortunately there is no out of the box feature to do this, so we need to edit our build xaml to do it. Lets get started:-
- Create a new Visual Studio 2010 class library project. This will be the container for our build xaml file. Add the following references :-
- Microsoft.TeamFoundation.Build.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0.
- Microsoft.TeamFoundation.Build.Workflow.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies.
- System.Drawing.dll which can be found at <Program Files (x86)>\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0
- Microsoft.TeamFoundation.VersionControl.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0
- Microsoft.TeamFoundation.WorkItemTracking.Client.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0
- Microsoft.TeamFoundation.TestImpact.Client.dll which can be found in the GAC at <Windows>\assembly\GAC_MSIL\Microsoft.TeamFoundation.TestImpact.Client\10.0.0.0__b03f5f7f11d50a3a
- Microsoft.TeamFoundation.TestImpact.BuildIntegration.dll which can be found at <Program Files (x86)>\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies
- Create a new build definition with a new build process template (xaml file) see here for more details.
- Add the xaml file created in the previous step to your Visual Studio project and open it.
- Locate and Expand the “Compile, Test, and Associate Changesets and Work Items Parallel” shape. Your screen should now look like this.
If you scroll down the workflow and find the “Get Impacted Tests, Index Sources and Publish Symbols” parallel shape you will see that the check for impacted tests happens after the run tests activity. This means that if we use the xaml out of the box all of the tests will run every time.
We need to reverse this order. If we can scope the Run Tests shape with the output of the Get Impacted Tests shape then we should be all good.
This just leaves one problem. At various points (maybe every night, maybe after each release) we need to gather the data for impact analyis by running all of the tests (to include any code changes since we last ran it) This should be configurable.
Right lets get started. Find the "If PerformTestImpactAnalysis" parallel shape as below:-Drag this shape upwards and insert here:-
Things should now look like this:-By moving this shape we have ensured that the Impact Analysis will occur before any tests are run. We now need to somehow scope the tests to only those that have been impacted by changes in the build. Fortunately we are in luck. The "GetImpactedTests" shape (msdn ref) has an output parameter called (funnily enough) ImpactedTests. This is defined as List<Test>
The "MSTest" shape (msdn ref) has an optional input parameter called TestNames. This is defined as IEnumerable<String>
All we need to do is enumerate through the list of tests returned by the GetImpactedTests shape and pass them as a list of strings to the TestNames parameter of the MSTest activity. The steps to do this are:-
- Create a workflow variable called ImpactedTestList of type TestList.
- Create a variable called TestNames of type TestList
- Select the "Get Impacted Tests" activity and set its ImpactedTests property to be the ImpactedTestList variable that we just defined.
We need to convert the TestList into a list of strings for MSTest. To do this we will use a ForEach shape to enumerate the List and an AddToCollection shape to build the list of strings. We can then pass this list to MSTest.
- Now grab a ForEach shape from the toolbox and place it directly after the "GetImpactedTests" shape. Set the DisplayName to "Create Test List".
- Enter ImpactedTest after Foreach and enter ImpactedTestList after in.
- Drag an AddtoCollection shape and add it to the body of the ForEach shape.
- Set the DisplayName of the AddtoCollection to Add Impacted Test to List.
- Set the Collection to be TestNames, the TypeArgument to String and the Item to ImpactedTest.TestName.toString()
- Your ForEach shape should now look like this:-
We need to pass this list of tests into the Run Test shape. To do this find the “Try Run Tests” Try Catch shape below our current position. Impact analysis only works when MSTest is run with a test settings file so we only need to update the Run Test activity in the middle. i.e. the middle red circle below.
Set the TestNames property of the “Run MSTest” shapes to our TestNames variable.
This just leaves us with two problems. Firstly how do we get all of the tests to run for the first build? Otherwise we will never have any impacted tests. The second problem is how do we ‘reset’ the impact data to include new code. We may want to do this nightly or every release/sprint etc. Fortunately we can solve both of these problems with the same solution. We need a way to override the scoping. If the TestNames list is empty then MSTest will run all of our tests. This solves our problem but introduces an edge case. What happens when there are no tests impacted by our code changes. We don't want any tests to run in that case.
We can solve this by using a custom process parameter. Follow the guidance here to add this to the Process tab of the definition editor. I called my parameter Is Base Line Run, and placed it in the advanced section. The parameter is a Boolean.
We can now add an If shape like this:-
The Condition property on the If shape is:-
TestNames.Count = 0 And BaseLineRun = False
In the “Then” portion of the shape we add an Assign shape and set DisableTests = True.
This will mean that if there are no tests in the TestNames List and it isn’t a base line run then no tests will run (because nothing was impacted) However if the base line run parameter is set to True then all of the tests will run.
Putting it all together
- Check your build xaml file in.
Right, time to queue a new build. Remember to set the Is Base Line Run parameter to True for the first build. This should run all of your tests. Here is the build output for my first build.
This ran all 7 tests in my solution.
Now lets change some code. Here is the Test Impact View in Visual Studio showing that my changes have impacted two tests. I now checkin my changes.
Now I queue another build, leaving the Is Base Line Run at False (the default.) The build output now shows:-
Success!! We could setup a nightly build with the Is Base Line Run parameter set to True and have out impact data being updated every night.
Link to final xaml is here