Blue Green Deployments with VSTS

Create an automated deployment process with no downtime, that allows sites to be tested on the live environment before being exposed to customers.

As this is a reference for CI/CD with VSTS, some sections contain lots of details not needed for a general process.

Prerequisites

VSTS Account set up
You code base is hosted in VSTS using git (though should work fine with github integrations or using tfs)
Deployment servers already exist, either local servers, or VMs in Azure.
Deploying a MVC site to be hosted in IIS

Future improvements

Add a proper load balancer
Replace IIS with self hosted .dotnetcore
Use Docker
Source control and centralise the powershell release steps

General tips

To enable the web.config to be checked in, but not cause constant merge clashes with other developers configs, the app and connection settings are broken out into separate files. The checked in version always pointing to development settings.

  <connectionStrings configSource="connectionStrings.Development.config" />
  <appSettings file="appSettings.Development.config" />

The build agent will then update this to point at production config files which are checked in with placeholder values.

Build Step tips

Updating web.config

Create a new powershell step. Set version to 2.* (to increase max characters) and 'Type' to inline. The following updates all web.config settings to production files.

$webConfig = "$(BUILD.SOURCESDIRECTORY)\**YOU_PROJECT_NAME**\Web.config"
$doc = (Get-Content $webConfig) -as [Xml]
$connectionObj = $doc.configuration.connectionStrings
$connectionObj.configSource = 'ConnectionStrings.Production.config'
$appSettingsObj = $doc.configuration.appSettings
$appSettingsObj.file = 'AppSettings.Production.config'
$rewriteRules = $doc.configuration.'system.webServer'.rewrite.rules
$rewriteRules.configSource = 'rewriteRules.Production.config'
$doc.Save($webConfig)

Use Yarn

For some reason npm install is terribly slow on VSTS, for now I recommend using Yarn, though future versions of npm may improve this.

Add 'Yarn Tool Installer' step. Add 'Yarn task' step.

NuGet restore

The default task caused me all sorts of issues, and switching to version 2.* broke the build as you need custom settings. See my answer below.

NuGet Restore 2.* config

Visual Studio Build

Make sure you set Visual Studio Version to the latest.

Example MSBuild Arguments for pre-compiled templates.

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\" /p:AspNetCompileMerge=true /p:PrecompileBeforePublish=true

Build Options

Build number format

$(date:yyyyMMdd)$(rev:.r)

Release Agent Setup

VSTS Agent Setup

Create a new user called TeamServicesDeploy (or as you prefer) on the server.

Give the user read/modify permission to folder C:\Windows\System32\inetsrv\config

Create a new Deployment Group https://YOUR_COMPANY_NAME.visualstudio.com/YOUR_TEAM_NAME/_machinegroup

When the new agent install powershell appears, tick User a personal access token, then copy to clipboard.

Open an administrator PowerShell on the server. Paste the powershell script in. At some point it will ask for username and password, use details you set up for TeamServicesDeploy on the machine.

Once this has run, inside Release on TeamServices, you can now select this server as a deploy location.

Make sure .Net 3.5 is installed.

Blue/Green IIS Setup

Use the Web Platform Installer to install ARR (Application Request Routing 3.0)

Create 3 websites in IIS under sites.

(YOUR_WEBSITE_NAME) - This is our fake load balancer

Add this site to your DNS, it will forward traffic to the below sites depending on which rule is enabled.

Create 3 bindings

http/80, *, your-website-url - Add this to your DNS
https/443, *, your-website-url - Add this to your DNS
http/80, *, stg.your-website-url - This is the url to your offline site, only add it to private company DNS

Select a physical path and create a web.config file. Make sure your TeamServicesDeploy user has read/modify permission to this file.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Staging/Live Load Balancer -->
<configuration>
  <system.web>
    <customErrors mode="Off" />
  </system.web>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="Proxy to Blue" enabled="false" stopProcessing="true">
          <match url=".*" />
          <conditions>
            <add input="{sites-blue:{HTTP_HOST}}" pattern="(.+)" />
          </conditions>
          <action type="Rewrite" url="http://{C:1}{REQUEST_URI}" appendQueryString="false" />
        </rule>
        <rule name="Proxy to Green" enabled="true" stopProcessing="true">
          <match url=".*" />
          <conditions>
            <add input="{sites-green:{HTTP_HOST}}" pattern="(.+)" />
          </conditions>
          <action type="Rewrite" url="http://{C:1}{REQUEST_URI}" appendQueryString="false" />
        </rule>
        <rule name="Default" enabled="true" stopProcessing="true">
          <action type="CustomResponse" statusCode="503" statusReason="Service Unavailable" />
        </rule>
      </rules>
      <rewriteMaps>
        <rewriteMap name="sites-blue">
          <add key="YOUR_LIVE_WEBSITE_URL" value="127.0.0.2" />
          <add key="YOUR_STAGING_WEBSITE_URL" value="127.0.0.3" />
        </rewriteMap>
        <rewriteMap name="sites-green">
          <add key="YOUR_LIVE_WEBSITE_URL" value="127.0.0.3" />
          <add key="YOUR_STAGING_WEBSITE_URL" value="127.0.0.2" />
        </rewriteMap>
      </rewriteMaps>
    </rewrite>
  </system.webServer>
</configuration>

(YOUR_WEBSITE_NAME)-green - Empty site that will contain deployed code

Add binding IP address 127.0.0.2:80

(YOUR_WEBSITE_NAME)-blue - Empty site that will contain deployed code

Add binding IP address 127.0.0.3:80

Release step tips - Blue Green

CD Triggers

Set up your Artifact with Continuous trigger

Set up your test environments with Pre-deployment conditions

After release trigger

Deployment Queue settings

1 number of parallel deployments
Deploy latest and cancel the others

As we are setting this environment up with blue/green, it doesn't matter that releases will constantly be pushed to it, as only a manual intervention by the developer will actually make it 'live'.

Deployment Steps Blue/Green

3 Steps:

Deploy code.
Manually check site runs ok
Swap blue/green

Deploy code step

Power Shell step - Blue Green

Queries IIS for currently live site (either blue or green) and sets a variable to the opposite to deploy to. (If blue is live, deploying to green). Creates a folder to deploy to, using the release number. Updates IIS settings setting the code location for offline site to this new deploy location.

$deployTo = If ((Get-WebConfigurationProperty "system.webServer/rewrite/rules/rule[@name='Proxy to Blue']" -PSPath "IIS:\sites\$env:IISSiteName" -Name enabled).Value) {"$($env:IISSiteName)-green"} else {"$($env:IISSiteName)-blue"}
Write-Host "##vso[task.setvariable variable=websiteName]$($deployTo)"

$deployFolder = "$($env:DeployLocation)$($env:BUILD_BUILDNUMBER)"
Write-Host "##vso[task.setvariable variable=templateLocation]$($deployFolder)\Templates"
Write-Host "##vso[task.setvariable variable=deployedWebsiteRoot]$($deployFolder)\"

mkdir -p $deployFolder

Import-Module WebAdministration
Set-ItemProperty "IIS:\Sites\$($deployTo)\" -name physicalPath -value $deployFolder

IIS Web App Deploy step

Update website name to $(websiteName).

Tick XML variable substitution

Remove Additional File at Destination

Powershell update additional web.config settings

XML variable substitution has its limits, so powershell is needed for some settings. Below updates SMTP settings and the error page to release variables.

$webConfig = "$(deployedWebsiteRoot)\web.config"
$doc = (Get-Content $webConfig) -as [Xml]

$httpErrorObj = $doc.configuration.'system.webServer'.httpErrors.error
foreach($httpError in $httpErrorObj)
{
    if ($httpError.statusCode -eq "500")
    {
        $httpError.path =  "$(errorPage)";
    }
}

$mailSettingsObj = $doc.configuration.'system.net'.mailSettings.smtp
$mailSettingsObj.deliveryMethod = "network"
$mailSettingsObj.network.host = "$(smtphost)"


$doc.Save($webConfig)

Manual Intervention Step

Create a new Agentless phase
Add a Manual Intervention step
This is when you run smoke tests against the offline site to make sure it is spun up ok with no config errors.
Add a Timeout of a few hours (e.g. 240 minutes). This is important as it rejects deployments that a left over from the day before.

Swap blue/green - Deployment Group phase

Skip downloading of artifacts.

Powershell Task - Switch Blue/Green

$state = (Get-WebConfigurationProperty "system.webServer/rewrite/rules/rule[@name='Proxy to Blue']" -PSPath "IIS:\sites\$env:IISSiteName" -Name enabled).Value
Start-WebCommitDelay
Set-WebConfigurationProperty "system.webServer/rewrite/rules/rule[@name='Proxy to Blue']" -PSPath "IIS:\sites\$env:IISSiteName" -Name enabled –Value (-Not $state)
Set-WebConfigurationProperty "system.webServer/rewrite/rules/rule[@name='Proxy to Green']" -PSPath "IIS:\sites\$env:IISSiteName" -Name enabled –Value $state
Stop-WebCommitDelay -Commit $true

Application Insights release annotation step

If you are using Application Insights, this is when you add your release tag step.