Tuesday, 12 December 2017

Creating a yaml CI Build for .net application

One of the great announcements from this year's Microsoft Connect() conference was YAML support for VSTS build definitions. 

For me, it's a great way forward towards "codifying" the build pipeline. The current TFS builds technology, introduced in Team Foundation Server 2015, despite all the benefits of a loose and extensible mechanism is rather difficult to maintain as code and doesn't really fit "pipeline as a code" definition. If you remember, earlier versions of Team Foundation Server (TFS 2005 and TFS 2008) used an MSBuild file to run builds. Whilst this was easy to code and maintain, extensibility was rather limited. Then, Team Foundation Server 2010 introduced XAML builds with better support for workflows but was difficult to work with. TFS 2015 simplifies XAML but the whole logic is spread across different aspects of the build. Yaml solves this shortcoming nicely.


Enable YAML Builds Preview Feature

At the time of writing this post, support for YAML builds is still in preview in VSTS. To enable it for your account, click on your profile and select option "Preview Features" from the drop down menu




Select option "from this account [projectName]". Scroll down till you find the "Build Yaml definitions" feature and set it to On



We are now ready to use YAML builds.

Creating a Yaml Build 

There are two ways in which we can set up a Yaml build. 

1) Create a file called .vsts-ci.yaml. When you push your change with this file to TFS, a build definition, using this file is created for you.

2) Explicitly create a build definition using the YAML template, providing the path of YAML file that you have committed in the repo.

We will go with option 2.

Create build.yml file


YAML file format is the format of choice for configuration files and is used by some exciting technologies like docker, Ansible, etc. It's great that VSTS now supports it as well. 

For this demo, we are creating build for a simple .Net Web Application. For the purpose of building our application, we need to do the following

1) Restore all NuGet packages
2) Build the entire solution

Our very simple yaml file looks as below

steps:
- task: nugetrestore@1
displayName: NuGet Restore
inputs:
projects: "MyWebApplication.sln"

- task: MSBuild@1
displayName: Building solution
inputs:
command: build
projects: "MyWebApplication.sln"


The file is pretty much self descriptive. As you can see we we have two tasks. The first task uses nugetrestore passing the solution as input. The second task executes MSBuild passing the applicatino's solution.

Commit the file to your local Git Repo and push to commit to TFS.

Creating the Build definition

Now that our yaml file is committed, we will create a build definition to use it. To do this, click on the New button to create build definition. For the build template, select YAML and click Apply




We will then be asked to provide the build name, agent queue and path to Yaml file. Make sure, you have selected the correct repository and branch in the "Get Sources" option for the build definition.

Please note that YAML builds are only supported for Git are not supported when TFVC is used a version control repository.

Click on the Triggers tab to make sure that Continuous Integration is selected as option

Click save to save your build. Now this build is set up as a continuous integration build for your repository and is triggered with every commit.


Conclusion:
If you compare the amount of work you had to do to create yaml build, it's really a breeze as compare to TFSBuilds. There are many use cases of using YAML for your build definitions. You can set up a complete pipeline, decoratively executing each steps which can be developed and tested locally before being used in VSTS.


Saturday, 2 December 2017

TFS 2017 Build - Partially succeed a build

At times, there is a need to explicitly set a Team build's result to be "Partially Successful". 

In Xaml, the way to forcibly build to set as partially successful is to set the build's "CompilationStatus" property to true and "TestStatus" to False, as shown below

<mtbwa:SetBuildProperties DisplayName="Set TestStatus to Failed so we get a PartiallySucceeded build" PropertiesToSet="TestStatus" TestStatus="[Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus.Failed]" />

Setting a TFS 2017 build to partially succeed is a bit more intuitive. Simply add a powershell task with an inline script and set the task's result to "SucceededWithIssues". Make sure it's the last task in your build, so that it doesn't affect the flow of task execution. The Powershell statement is shown below

Write-Host "##vso[task.complete result=SucceededWithIssues;]DONE"

My build looks as follow


Thursday, 30 November 2017

TFS 2017 Build System - Maintain last "N' builds

In my last blog post, I described retention policies in the TFS 2017 build system. I described how different it is from the the retention policies we get in XAML build system. 

One of the limitations I found in the new style retention policy is that I couldn't retain a specific number of builds for each status. We needed to do it for some builds that are triggered very frequently (once every couple of minutes) and check if there is some work to be done. If it found work, it would do it, otherwise it will reschedule a build for itself after a couple of minutes. Another scenario, where you might have a lot of builds is when you it is triggered by a commit of a very busy repository.

So, in order for us to retain only "N" builds for each status, we created a Powershell Module to clean up builds. In the module, we create a command-let that takes as parameter the name of the build, the number of builds to keep, the result filter and tag filter. Our command-let looks as following



***************************************************
.SYNOPSIS
 Cleans up all builds for the given build definition keeping the latest N number of builds where n is passed a parameter
 If a status is provided, it would only keep N builds with the given status

.DESCRIPTION
 PATCH https://{instance}/DefaultCollection/{project}/_apis/build/builds/{buildId}?api-version={version}
 Uses api-version 2.0 to update the build result
***************************************************
function Cleanup-Builds([string] $tfsCollection,
                    [string] $tfsProject,
                    [string] $buildDefinitionName,
                    [int] $numberOfBuildsToKeep = 10,
                    [string] $result="",
                    [string] $tagsFilter = "")
{
    if (${env:system.debug} -eq $true) {
        $VerbosePreference="Continue"
    }

    if ($status -eq ""){
        Write-Verbose "Deleting all but the latest $numberOfBuildsToKeep builds for definition $buildDefinitionName."
    }
    else{
        Write-Verbose "Deleting all but the latest $numberOfBuildsToKeep builds for definition $buildDefinitionName with status $status."
    }

    $buildDefinition = Find-BuildDefinition($buildDefinitionName)
    if ($buildDefinition -eq $null) {
        Write-Error "No build definition found $buildDefinitionName"
        return
    }

    $buildDefinitionId = $buildDefinition.id
    $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds?api-version=2.0&definitions=$buildDefinitionId&queryOrder=2&resultFilter=$result&tagFilters=$tagsFilter&`$top=5000")

    $builds = Invoke-RestMethod -Method GET -UseDefaultCredentials -ContentType "application/json" -Uri $query
    $retainedBuild = 0
    $deletedBuildCount = 0
    for ($i = $builds.Count - 1; $i -gt -1; $i--) {
        $build = $builds.value[$i]
        $buildId = $build.id
        $buildNumber = $build.buildNumber
        
        try {
            $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds/$buildId/tags?api-version=2.0")
            $tagFound = $false

            # Not delete the latest numberOfBuildsToKeep builds
            if ( ($retainedBuild -lt $numberOfBuildsToKeep)) {
                $retainedBuild = $retainedBuild + 1
            }
            else {
                Write-Verbose "Deleting build $buildNumber"
                $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds/$buildId`?api-version=2.0")
                Invoke-RestMethod -Method DELETE -UseDefaultCredentials -ContentType "application/json" -Uri $query
                $deletedBuildCount = $deletedBuildCount + 1
            }
        }
        catch {
            Write-Error "StatusCode:" + $_.Exception.Response.StatusCode.value__ +
                        "`r`nStatusDescription:" + $_.Exception.Response.StatusDescription
        }
    }
        
    Write-Output "Deleted $deletedBuildCount builds for build definition $buildDefinitionName"
}

We create a PowerShell Module file for the above command let. To set up the Powershell module, we modified the PSModulePath environment variable as first step of our build to include the module path. Then to set-it all up we added a PowerShell task group calling the Cleanup-Builds command in an inline script as shown below




Our build definition looks like below





Friday, 10 November 2017

Retention Policies for TFS 2017 Build System

TFS build system has had a major overhaul since TFS 2015. For people working on team builds since TFS 2010, there is some major learning curve. One of the things the people often find confusing is the retention policy in the new build system. In earlier versions of TFS, you could specify how many builds you want to retain for each status as shown in the screenshot below


Retention Policies for Xaml Builds

The retention policy is quite obvious and you have a deterministic number of builds retained at each status. It's not quite the case in the new build system. A sample retention policy in the new system looks like following


Retention Policies for TFS Builds


So what does it mean? 

Well, to say it simply, it means exactly what it says on the tin!! In the example above, the build would keep all builds from the last 4 days and keep a minimum of 20 builds. That is if there are less than 20 builds present for the given build definition, it would keep older builds until there are a minimum of 20 builds. Lets ignore the options with the lock sign, we will come back to it later. Note there is are no maximum count. It means that you can't control how many builds you keep for your build definition. This is a major shift from earlier retention policy where the number of builds kept for a build definition was deterministic. 


When are builds actually cleaned up?

If you are using an on-premise version of TFS (I am using TFS 2017 Update 2), the builds are actually not cleaned until 3:00 AM every day. For VSTS, it happens several times in a day but the time is not deterministic. The actually explains why there is only a "Minimum to Keep" option in the retention policy.

If you have a build definitions that is triggered very frequently, you will need to find a solution of actually deleting the build definitions. I will explain it in the next post.


What about Keep For 356 days, 100 good builds?

This is the option you see below your policy in the screenshot shown above. This is in fact a TFS Project Collection wide policy and enforces the maximum retention policy. So, in the example above, you can't set "Days to Keep" to more than 365 and "Minimum to Keep" to more than 100. In fact, if you have appropriate permissions, you can change it for the entire Team Project Collection.



TFS Project Collection Retention Policy


Multiple Policies

If you want, you can add retention multiple policies for your build definition. It is very useful, if you have build definition that builds different code branches (release branches for instance). You can use the retention policies to keep different number of builds from each branch. 


Multiple Policies


If you have multiple retention policies for the same branch, the retention would be the most lenient of all the retention, so whatever retains the most builds.

In the next blog post, I will show how we are keeping a lid on the number of builds for builds which are build very frequently, every couple of minutes in our case.

Sunday, 22 October 2017

C# Async Programming - Tasks for dummies

There are umpteenth articles / blogs / guides about the Task-based Asynchronous Pattern used in C# for asynchronous programming. However, I feel that explanations are often convoluted and difficult to follow for something new to language / programming. This week, I explained the pattern to a graduate following some review comments and couldn't find an easy-to-understand article, so thought to explain it myself here.

This will be a series of blogs. I will try to keep it as simple as possible without compromising on completeness. Starting with one of the most basic concept - "The Task".

What is a Task?
Task is the C# abstraction of an asynchronous operation. In other words, it some series of code that executes asynchronously. It may or may/not return a result. 

How are Tasks created?
Tasks can be created explicitly by creating an object of type Task or Task or implicitly by running an async method. For example, both of these expressions end up resulting a task

var task = Task.Run( () => { … });

Func taskFunction = ( async () => { await foo() } ); taskFunction.Invoke();

Tasks and Threads
There are some differences when tasks are created explicitly or implicitly but let's not go there. 

There is a common misunderstanding that creating a task means running on a new thread. This is not true. Whether or not task creates a new thread depends upon how it is created.

For tasks created using Task Parallel Library, using Task.Run() for instance, a thread is created with the task. Running the following code 

              
Console.WriteLine($"Application Thread ID : {Thread.CurrentThread.ManagedThreadId}");
Task.Run(() =>
{
Thread.Sleep(30);
Console.WriteLine("Inside Task");
Console.WriteLine($"Task Thread ID : {Thread.CurrentThread.ManagedThreadId}");
});

Will produce

Application Thread ID : 2
Back to application Thread ID : 2
Inside Task

Task Thread ID : 3

Tasks created by async methods DO NOT create a new thread. Once once task is blocked, control is shifted to other task that is in ready state. 

For example, the following code 

Console.WriteLine($"Application Thread ID : {Thread.CurrentThread.ManagedThreadId}");
Func localTask = (async () =>{Console.WriteLine("Inside Task");Thread.Sleep(30); 
Console.WriteLine($"Task Thread ID :{Thread.CurrentThread.ManagedThreadId}");
});
localTask.Invoke();               
  
Will produce 

Application Thread ID : 2
Inside Task

Back to application Thread ID : 2
Task Thread ID : 2

Note that the thread Id is the same. This means that tasks, unless created by task parallel library, do not run in parallel. They share the same thread and uses context switching to pass control as tasks are blocked and become available again.

The Await Operator
This brings us nicely to the await operator. In simplest words, the await operator cause context switching. The operator is used when executing code block needs to get a result from a task that is running asynchronously. Calling await will block the current routine, only to be returned when the task it is waiting on has completed.

Conclusion
I hope this post will help people in understanding C# tasks. Some of the key take away from this post
  • Tasks can be explicitly using Task Parallel Library or implicitly using async keyword.
  • Task are not same as threads. Some tasks are created in a new thread - the ones created by TPL for instance. While others are created on the same thread.

Sunday, 15 October 2017

Visual Studio 2017 - New npm package won't install...

One of my "how I got burned today" blogs. Spent some time on it so thought to share.

I started with writing a simplistic application using NodeJS today using Visual studio 2017. Tried to install NodeJS using the "Install new npm Package option".



The "Install New npm package" dialog opens. Typed in the name of the package and clicked install "Install Package"


Absolutely nothing happened. No errors or messages were shown and my package wasn't installed. Turned out that there was a syntax error in my packages.json file, where I had missed out a comma. Would have been nice if visual studio had captured this error and shown some sort of message. I am using Visual Studio 2017 Update 3.

Friday, 13 October 2017

PowerShell - The curious case of @ in converted Json strings

PowerShell is great when it comes to working with JSON. Being a scripting language, you can pretty much de-serialize your json without declaring types for them, do your work on de-serialized objects, and then serialize it back to for storing or transport. 

The ConvertFrom-Json and ConvertTo-Json are powerful functions. However, there are a few nuggets that you need to be aware of. I got caught out by one of the so thought to blog about it.

When working with ConvertTo-Json, be mindful of the -Depth parameter. The parameter specifies how deep it should go in our object while converting to json string. The default value is 2. What it means is that if you have a complex json object that goes down more than two levels of depth and it you haven't specified the -Depth parameter, your nested objects would be treated as Hashtable.

As an example, let's assign a json string to a variable 

 $programmersJson = '[{ 
      "Name" : "Hamid",
      "Gender" : "Male",
      "Expertise": [  
        {
          "Skill": "PowerShell",
          "Level": "5"
        },
        {
          "Skill": "C#",
          "Level": "8"
        }
      ]},
      {
      "Name" : "Adnan", 
      "Gender" : "Male" ,
      "Expertise": [
        {
          "Skill": "PowerShell",
          "Level": "7"
        },
        {
          "Skill": "C#",
          "Level": "6"
        }
      ]}]'

The json string contains an object that is three level deep. . The top level object is a collection, each item in the collection is an object with a "Name" and "Gender" property as well as a collection of objects each has a "Skill" and "Level" property

Now, lets call ConvertFrom-Json

$programmers = ConvertFrom-Json -InputObject $programmersJson

Write-Output $programmers

The result is an array of objects, as expected

Name  Gender Expertise
----  ------ ---------
Hamid Male   {@{Skill=PowerShell; Level=5}, @{Skill=C#; Level=8}}
Adnan Male   {@{Skill=PowerShell; Level=7}, @{Skill=C#; Level=6}}

Now, lets try to convert it back to json. So, when you call

 ConvertTo-Json -InputObject $programmers

You would expect the Json string to be same as $programmersJson. Wrong!! The string you get back is


[
  {
    "Name": "Hamid",
    "Gender": "Male",
    "Expertise": [
      "@{Skill=PowerShell; Level=5}",
      "@{Skill=C#; Level=8}"
    ]
  },
  {
    "Name": "Adnan",
    "Gender": "Male",
    "Expertise": [
      "@{Skill=PowerShell; Level=7}",
      "@{Skill=C#; Level=6}"
    ]
  }
]

Notice, the @ sign for each of the item in the Expertise collection. It means that the function has treated each item as a Hashtable rather than an object.

Now execute the following

 ConvertTo-Json -InputObject $programmers -Depth 3

The depth parameter will make it treat each item of Expertise collection as an object as well and the resulting Json would be as you would expect.

[
  {
    "Name": "Hamid",
    "Gender": "Male",
    "Expertise": [
      {
        "Skill": "PowerShell",
        "Level": "5"
      },
      {
        "Skill": "C#",
        "Level": "8"
      }
    ]
  },
  {
    "Name": "Adnan",
    "Gender": "Male",
    "Expertise": [
      {
        "Skill": "PowerShell",
        "Level": "7"
      },
      {
        "Skill": "C#",
        "Level": "6"
      }
    ]
  }
]


This is how we would have expected the output json string to look like. So, next time you are working with json on PowerShell, make sure to be mindful of the -Depth parameter.