Monday, April 25, 2011

Team Build–Enhancing the Build Deploy Test Workflow

Microsoft provided an unofficial unsupported xaml to enable automation of the build-deploy-test cycle for physical machines.  This mimics the functionality provided by Lab Manager for Hyper-V virtual machines.  I used this in a previous post to Integrate with FitNesse.

One of the limitations of this build is picking the latest successful build given a certain build definition.  The xaml file supports queuing a new build or picking the build results  from a share, but not picking the latest build, given a build definition.

Well I decided that I wanted to use the latest build from a given definition so I have updated the xaml to do that for us.  Here is how I went about it.

The Approach

We will create a custom code activity that retrieves the Last Good Build of a passed in Build Definition name.

We can then update the BuildDeployTest_Physical.xaml to use our new activity.  We can pass this build into the rest of the workflow.

Getting Started

Make a copy / branch of the BuildDeployTest_Physical.xaml file.

Create a new build definition that uses your new xaml file.

Create a new C# Project and include your xaml file as a link.  Add the references needed in order to get the project to compile.

Create a new C# Class Library – this will be our custom code activity.  My code for the class looks like this:-

[BuildActivity(HostEnvironmentOption.All)]
   public sealed class RetrieveLastGoodBuildNumber : CodeActivity<string>
   {
       private IBuildServer buildServer;

       public InArgument<IBuildDetail> CurrentBuild { get; set; }
       public InArgument<string> BuildDefinitionName { get; set; }
       public OutArgument<string> BuildDropLocation { get; set; }

       protected override string Execute(CodeActivityContext context)
       {
           buildServer = this.CurrentBuild.Get(context).BuildServer;
           string projectName = this.CurrentBuild.Get(context).TeamProject;
           string BuildDefName = this.BuildDefinitionName.Get(context);
           IBuildDefinition def = buildServer.GetBuildDefinition(projectName,BuildDefName);
           IBuildDetail buildDetail = buildServer.GetBuild(def.LastGoodBuildUri);
           this.BuildDropLocation.Set(context,buildDetail.DropLocation);
           return buildDetail.BuildNumber;
       }
   }

Note the BuildActivity attribute at the top of the class.  This ensures that this gets loaded as part of the build.  I have used two arguments that get passed in.

CurrentBuild is a reference to the IBuildDetail of the currently executing build.  This enables us to work out the current Project.

BuildDefinitionName is the Build that we want to find that last good build to deploy and test against.

There is one out paramater:-

BuildDropLocation is an out parameter that gives us the drop location of our last good build.

The return string value is the Build Number of the last good build that matches the build definition passed in.

Putting it all Together

Open your xaml file, go to the Build section and insert an IF shape where you see the red arrow below.

image

We will add a new argument called GetLatestGoodBuild (bool) to the build workflow and metadata

 image

This will be the trigger to use our new code activity.   (If both DoBuild is True and this parameter is True then it will use the last Good build of the Build Definition Name parameter.

We set the IF Condition to be GetLatestGoodBuild = False.  Then we can add the following:-

image

The Then branch does the Build as in the original xaml.  The Else branch implements our new behaviour.  Add a sequence called Find Latest Good Build and then drag our new custom code activity (which should be already in the toolbox.)  Set the parameters like this:-

image

Finally I have added a WriteBuildMessage shape to output the build number that we have found.  NB I have set the Importance to Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High so that it always displays in the log file.

Deployment

Check in the xaml file and deploy the custom code activity.  More info on how to do that here.  Check out the “Deploying the Activity” Section.

The Results

Fill in the parameters (make sure to set both the Queue New Build and Get Latest Good Build to True) and queue a new build.  My result looks like:-

image

Here is the log file

image

Success, by only specifying a build definition we have found the latest good build, deployed the application from the build drop folder and run our tests (with the test run being associated with the correct build)

As usual all code from here.

Sunday, April 17, 2011

Visual Studio SP1 Issues

If you get the following error

(QTController.exe, PID 2864, Thread 17) ControllerExecution: Wrong run state during ReverseDeploymentComplete: Microsoft.VisualStudio.TestTools.Exceptions.EqtException: Expected state Completing, but actual state is Disposed

at Microsoft.VisualStudio.TestTools.Common.StateMachine`1.Verify(StateEnum expected)

at Microsoft.VisualStudio.TestTools.Controller.ControllerExecution.ReverseDeploymentComplete(Guid runId, FileCopyFinishedReason reason)

Then you have sp1 installed on your Test Controller machine but not on the Test Agent machine.

Thursday, April 14, 2011

Fail a Build if Code Coverage is Low

UPDATE: I have included the Code Coverage exe in the download

I have seen lots of questions about how to manage code coverage. Some people use Custom Check In policies to stop people checking in code with low coverage. This is one way to do it. Traditionally these have been difficult to deploy and manage although it getting better. Another option is to fail the build if the code coverage drops below a certain threshold. This has the advantage of being a server side solution and therefore doesn’t rely on anything special installed on the client.

Approach

We can build on this approach that was used in Visual Studio 2005. The concept is the same. We will call out to an exe (using the trusty InvokeProcess activity from our build xaml file.) This executable will parse the binary data.coverage file that gets generated by MSTest. We can then sum up the Code Coverage to an overall percentage and compare it to a value in the build definition. If our coverage is too low we will fail the build.

Update the Code

The same I linked to above was based on VS2005. Things have changed a bit since then. Here is a good blog post on code coverage changes. I followed that post to ensure that the code will work with VS2010. I also changed the code to work out the total code coverage for the solution and output that to the console stream.

The Build

First lets create a new build definition.

image

We will then create a new Build Process Template

image

and open it. Add a new Argument to hold the code coverage limit.

image

Then click the Metadata argument add our percentage as a Process Parameter.

image

Click Add and enter Parameter Name and values.

image

This will be our code coverage limit that the build will accept. Anything less than this and we will fail the build. Now lets edit our Xaml file to use this value.

Add another variable to hold the total code coverage

image

We will need to add an InvokeProcess shape to our build process. I have added this inside the Run Tests sequence.

image

The total code coverage (as received from stdOutput) gets assigned to the CodeCovered variable.

The FileName is

image

obviously the CodeCoverageConverter.exe needs to be installed onto the Build Server in this path.

The arguments are set as follows:-

"/infile:" + TestResultsDirectory + "\CodeCoverageTests\In\" + System.Environment.MachineName + "\data.coverage /outfile:" + TestResultsDirectory + "\coverageresults.xml /exepath:" + TestResultsDirectory + "\CodeCoverageTests\Out\"

If you look at this you will see an interesting point. The default path for the TestResults In and out folder have been changed. These folders contains our .coverage file as well as the dlls used to execute the tests. The standard path includes the Date and Time that the test executed. In order for us to be able to find these files from the build we need to set the path.

Setting the Path

We can do this from our TestSettings file that the build uses. You can see from the Build process parameters here that my build is the local.testsettings file.

image

I change this as follows:-

image

This means that the TestResults will always be placed in that folder. However this introduces another issue. If we leave it like this I will end up with the following structure:-

TestResults

CodeCoverageTests

CodeCoverageTests[1]

CodeCoverageTests[2]

and so on

So as part of my build I need to delete this folder to ensure that there will only be one copy. To do this I have used a DeleteDirectory shape.

image

The parameters are:-

image

This will ensure that only one CodeCoverageTests folder exists.

We now need to check if the overall code coverage is below our threshold. To do this I have used.

image

The IF shape checks if our coverage is lower than our threshold. If it is then it will update the Build Test Status to failed and write a message to the Build Log as a Build Error.

Testing the Build

I have set the Code Coverage threshold at 80%. I have also selected the “Fail build on test failure” option

image

image

As usual I have included all the code as a sample. Running a build with 100% code coverage gives us.

image

No surprises there. If we now comment out some tests to bring out Code Coverage down to 50% we see.

image

Success we have failed our build!

Code can be found here.