Terraform in Azure DevOps Pipelines

The following will walkthrough deploying Azure resources using Terraform in Azure DevOps Pipelines. Although its a detailed guide, there is an expectation that the person following already has a good understanding of Git, Azure and Terraform.

It’s setup with the master or main branch created in DevOps and then cloned locally. A new branch is created and then committed to DevOps (Branch policies will restrict committing directly into master/main). A Pull Request is then created and the first pipeline is run which initialites a terraform plan and terraform validate. After the pull request is approved a build pipeline will run which initiates a terraform plan to a plan file, which is then archived as an artifact. If all checks are passed the release will need to be approved, which then starts a release pipeline. This performs a terraform apply using the plan file artifact, to deploy the resources into Azure. Once all is deployed the final step is to switch back to the master/main branch in VS code, pull the merged code and delete the old branch.

For a detailed video walkthrough, follow Terraform And Azure DevOps - How To Configure by Jack Tracey. I found this really helpful when setting up for myself.

Requirements

• Visual Studio Code
• Git
• Azure Subscription - Sign up for free credits
• Azure DevOps Organisation - Get one for free
  • Request free parallelism - If using the free plan

You now have to make a request for free parallelism if using private repos on the free plan as stated on Microsoft Docs:

Note We have temporarily disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. However, you can request this grant by submitting a request. Existing organizations and projects are not affected. Please note that it takes us 2-3 business days to respond to your free tier requests.

All code from the this post can be found on GitHub

Setup DevOps Project

I started by creating a free DevOps organisation and then creating a new private project called devops-terraform-build. Create a .gitignore file using the Terraform template which will basically tell it to not upload state and tfvars files. This created a repo under Repos which I cloned into VS Code.

Ensure you select a local directory and that you cd into the directory using your terminal, which has Git installed.

Azure Resources

A few things need to be in place in Azure in order for Azure DevOps and Terraform to be able to interact with it. Manually create the following in Azure to host your Terraform backend, SPN and secrets:

• A Resource group for the following resources:
  • Storage account and container for remote state
  • Key Vault
• An App registration, copying the relevant secret, application id, tenant id.
  • Assign the app registration Owner permissions on the subscription
  • Assign an access policy for the app registration of list and get, Secret Permissions in Key Vault

Open Key Vault and create the following new secrets:

Configure Project

Create a Service Connection in DevOps project to allow access to the Azure Subscription.

Select Project Settings > Pipelines / Service Connections > New Service Connection

Select Azure Resource Manager > Next > Service principal (manual) > Next

Configure it with the following:

Click Verify

Next, to create an Azure DevOps Variable Group, select Pipelines > Library > + Variable group

Name it the same as the Azure Key Vault which has been created. Select Link secrets from an Azure key vault as variables. Then select the Azure Subscription spn and the Key vault it has access to. Then finally add all of the secrets in the key vault.

Create the Plan CI Pipline

We’ll start by creating the CI/Build pipeline, but this isn’t the first pipeline to be initiated in the process. It’s easier to create this one first and then clone it later.

Click Pipelines > Create Pipeline > Use the classic editor. Name it terraform-plan

Select Azure Repos Git as the source and then ensure it has selected the project, Repo and branch. Click Continue

Click Start with an Empty Job. Rename the pipeline and select the “Agent Specification” as Ubuntu 20.04

Select Agent job 1 and rename it to the same name as the Pipeline. Then click the + to add a task. Search Terraform tool installer, and click Get it free. Install it, and select your organisation when prompted. Once installed enter the Version on the task as 1.2.6 (This was the latest version for linux at the time of writing).

Add a new Command Line task and name it terraform init. Set the script as:

terraform init -backend-config="access_key=$(Name of access key variable)"

This will tell it to check the variable group to match the stgAccountName-key-1 variable, which is a Storage Account’s access key.

Clone the task and this time rename to terraform validate. Set the script the same:

terraform validate

Clone it again and this time rename it to terraform plan and set the script as:

terraform plan -input=false -out=tfplan -var="spn-client-id=$(CHANGEME-spn-client-id)" -var="spn-client-secret=$(CHANGEME-spn-secret)" -var="spn-tenant-id=$(CHANGEME-spn-tenant-id)"

This will tell it not to prompt for any input variables at runtime and out to a tfplan file. Ensure the variables are updated with the name of your variables.

Now create a new task to create an archive of the plan file. The task to add is called Archive Files and configure as follows:

And finally it is time to publish the pipeline artifacts. Add a task called Publish Pipline Artifacts and configure as follows:

The pipeline tasks should look as follows:

That’s the last “task” to add for this pipeline. Now at the top of the pipeline, click Variables > Variable groups > Link variable group. Then select the variable groups which was created earlier.

Finally, click Triggers > tick enable continuous integration > Add the path terraform/ so that it only takes affect on git changes within the terraform directory (yet to create locally).

Click save when complete to save the pipeline.

Create the Status Check & Plan, Pull Request Pipline

We now need to create a new piplline, not for CI but for to validate the code after Pull Requests are created. Start by cloning the terraform-plan pipeline and make the following changes:

• Rename it to terraform-status-check-plan-and-validate
• Delete the last two tasks archive terraform plan files and publish terraform plan artifact
• Click the terraform plan task and delete the -out=tfplan from the Script.
• Click Triggers and un-tick (disable) Enable continuous integration
• Save the pipeline

Release Pipeline

The CD/release pipeline is the final one to configure to deploy the resources in Azure using Terraform.

Go to Pipelines > Releases > New pipeline > name it terraform apply > select Empty job.

Name Stage 1 as terraform apply. Then click Add artifact and select from the terraform-build pipeline which was created earlier.

Then enable CD by clicking the lightening icon, then Enabled under Continuous deployment trigger.

Configure the job, under Stages, click 1 job, 0 task. Enter the following:

Click the + to add a task. Search for and select Extract files. Change Archive file patterns to:

$(System.ArtifactsDirectory)/_terraform-plan/$(Build.BuildId)-tfplan/$(Build.BuildId)-tfplan.tgz

Set the Destination folder to:

$(System.DefaultWorkingDirectory)/

Add a new task called Terraform tool installer, set 1.2.6 as the version.

Add a new task called Command line, name it terraform init and use the same command line as in the build task.

terraform init -backend-config="access_key=$(Name of access key variable)"

Then Advanced > Working directory set it to:

$(System.DefaultWorkingDirectory)/terraform

Clone the Command line task and create a new one called terraform apply. Script is:

terraform apply -auto-approve -input=false tfplan 

All tasks will then looks as follows:

Now link our variable groups. Click Variables > Variable groups > Link varibale group > Select the variable group created earlier.

That’s all that is required to configure the pipeline tasks, but we need to add a Pre-deployment condition, go back to Pipeline and click the following:

Enable Pre-deployment approvals, I set myself as the approver as I am the only person in my DevOps organisation. I also changed the timeout to 7 Days.

The release pipeline should now look as follows:

Click Save

Branch Policies

The Branch policy will ensure that code is only being commited via a pull request and not directly in the main/master branch.

Click Repos > Branches > under your default branch, hover over it with mouse and in the far right, click the three dots > More options > Branch Policies. Set as follows:

Scroll down and enable Squash merge

Build validation > + (add). We only want the terraform-status-check-plan-and-validate pipeline to run if there have been any commits on *.tf files. Set as follows:

And finally add myself under Automatically included reviewers

And that’s it for the DevOps configuration. Now it is time to write some Terraform configurations and commit the code to test the pipelines.

Testing the Build and Release Pipelines

To create a new branch in VS Code, click Source Control > three dots > Branch > Create Branch > name it test

I should not be allowed to commit code changes directly to the master branch now there is a “branch policy” in place. Start by creating a folder called terraform and add the following .tf. files.

devops-terraform-build
│
│   .gitignore
│   README.md
│
└───terraform
        backend.tf
        providers.tf
        resource_group.tf
        variables.tf

The .gitignore will look something like this:

#  Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# .tfvars files
*.tfvars

The backend.tf file. Ensure you set the correct name for the resource group and storage account which will host the state file.

terraform {
  backend "azurerm" {
    resource_group_name  = "NAME OF RG"
    storage_account_name = "NAME OF STG"
    container_name       = "devops-tf-state"
    key                  = "terraform.tfstate"
  }
}

The providers.tf file. Ensure you set your unique subscription id.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.0.0"
    }
  }
}

provider "azurerm" {
  subscription_id = "SUBSCRIPTION ID"
  client_id       = var.spn-client-id
  client_secret   = var.spn-client-secret
  tenant_id       = var.spn-tenant-id
  features {}
}

The resource_group.tf file. Set the name of the resource group you want to create. This is all that will be created for now.

resource "azurerm_resource_group" "rg1" {
  name     = "devops-terraform-rg1"
  location = "UK South"
}

The variables.tf file.

variable "spn-client-id" {}
variable "spn-client-secret" {}
variable "spn-tenant-id" {}

Deploy Resources

Now we can deploy the resource group. In VS Code, click Source Control > then the + (stage) > Tick to commit > Name the commit > and then Push

Back in Azure DevOps, in Repos you’ll see the test branch has been commited and that you have an option to Create a pull request. Click it.

Fill in name and description and click Create

Notice the pipeline which performs a terraform plan and validate has passed

Click it to see the jobs.

Select the terraform plan job to see detailed output. Notice one resource will be created; the resource group.

If I open Azure and browse to the Storage Account and Container, I can see the state file created by the terraform init command.

Now approve the Pull request to merge the test branch into Master

This will initiate the build pipeline to create a plan file as an artifact.

Select archive terraform plan files for detailed output of the command.

Now approve the release by going to Pipelines > Releases

Then watch the release pipepline progress and deploy the Resource Group to Azure.

Once complete you can see the new Resource Group “devops-terraform-rg1” was created.

That’s a lot of work to deploy a resource group. But now I can create new branches with more resources and have Azure DevOps deploy then when the branch is merged. But first I will show you how to cleanup up the old branch in VS Code.

Clean up local repo

Now the “test” branch has been merged into main/master (in my example; master), we can cleanup the local test branch and swtich back to master.

git checkout master

You’ll notice in VS Code in the bottom left that the branch has changed.

Fetch and prune. The -p option tells fetch to delete any references that no longer exist on the remote origin. Git remote prune will also remove deleted branches.

git fetch -p

Then “pull” the updates on the origin repo to your local repo.

git pull

Delete the old, local “test” branch

git branch -D test

Verify your local branches

git branch -a

That’s it. Create a new branch if you want to commit new code as a pull request.