(Yet) Another Take on Integrating Terraform with Argo CD

Terraform is a popular tool for creating the underlying infrastructure for cloud-native applications. This includes Kubernetes clusters, databases, Identity and Access Management roles, etc (aka “cloudy stuff”). However, it is challenging to integrate the outputs of Terraform resources to be used with Argo CD.

Argo CD and Terraform

There have been a few attempts to bridge (pun intended, you’ll soon see why) these two worlds. The GitOps Bridge Pattern is one such notable way. There have been a few talks about it, and there is even a full blown implementation from AWS.

The gist of that pattern is that all values that your Argo CD Application needs, are exported from Terraform as outputs. These outputs need to be placed into an Argo CD cluster secret as labels or annotations (how you do that with Terraform is an exercise for the reader, but one could e.g. use the argocd_cluster resource with the community-driven Argo CD Terraform Provider). In order to consume the cluster metadata from an Argo CD Application, you need to use an ApplicationSet. More specifically an ApplicationSet using the cluster generator.

The gist of it would look something like this:

However, there are a few limitations to this approach:

• The obvious one is that you need to use an ApplicationSet. I’m a huge proponent of using ApplicationSets because it can easily template applications for dozens (or more) clusters. Sometimes you just might want to use a simple Application, or use an ApplicationSet with a different generator than the cluster one.

• It’s crucially important to note that with this approach, Git is no longer the only source of truth. In order to figure out what’s going on, you’ll need to look in Git, but also figure out where and how the metadata is being populated. This breaks the whole principle of GitOps: Git alone should be the source of truth.

• Getting those values into the application implies using inline Helm values. This can balloon quite rapidly - in the real world this can lead to having dozens of (or even more !!)  values. See the previous point.  

• Not Kustomize-friendly. There’s no obvious way to use Kustomize with this approach, which is why we are using Helm values here.

Can we do better?

Argo CD and Terraform - a Better Approach

Since the GitOps Bridge pattern was conceived, there have been a few developments. One of those being the advent of multi-source applications in Argo CD. This is useful when templating Helm applications - values can now be put into a Git repository while the Helm chart itself can be hosted in a Helm repository. While not strictly required, it’s more convenient than creating an umbrella chart hosting the values file(s).

The more important part comes below.

DISCLAIMER: This assumes the use of GitHub - I have not tested this with other SCMs, but I am aware of e.g. a GitLab provider. I can’t say if what I’m about to demonstrate works with other providers. 

As it turns out, as I was only recently made aware, there is a GitHub Terraform Provider. Even more 🤯, this provider can be used to create PRs and commit to a Git repository!

Armed with this knowledge, we can have our cake and eat it too. We can have Git as our source of truth, and get our Terraform values into Argo CD. How do we do that?

First off, you’ll need to set up the GitHub provider with a Personal Access Token (it needs repo scope).

For this exercise we’ll use a multi-source Helm application, but this can be used with an umbrella chart or even a Kustomize application. As long as we can commit to Git we’re good.

First off, let’s create some example values. It doesn’t really matter what these values are for this exercise.

The secret here is that the GitHub provider has a resource called github_repository_file. With this, we can create any content we want, as long as it’s a string. We can set an arbitrary path and branch in this resource, and if we set overwrite_on_create to true , any Terraform state changes will perform a commit if the file already exists.

Here’s an example:

Let’s do terraform apply. This will have made a commit to the tf-test GitHub repository with the values which have been templated from Terraform. When doing this within a GitHub organization, you will need to prefix the repository with your GitHub organization.

Now we can make use of it in an actual Argo CD Application.

Once applied, the Argo CD Application will make use of the values that have been templated from Terraform 🎉

So far, so good, but you probably don’t want to YOLO whatever Terraform spits out straight into `main` do you? (in case you do, please disregard the following section)

Luckily for us, the GitHub provider also provides a github_repository_pull_request resource. With this, any changes we wish to propagate will go through a PR first.

First off, we‘ll need to create a Git branch that changes with the content you wish to apply. For the sake of simplicity (sadly, not so much for the beauty), I’ve added a github_branch resource with a branch name that changes if the contents of the Helm values change.

We then use the branch name as an input to github_repository_file.

Finally, we’ll create a github_repository_pull_request resource, which opens a PR whenever the contents of the Helm values change.

Once again, terraform apply it and a PR will be created, which you can then merge at your convenience. If you change the Helm values and run terraform apply before the PR has been merged, the feature branch along with the PR will be deleted and a new PR and feature branch will be created.

What about Kustomize?

The good news is that with this variant of the GitOps Bridge pattern, we can also support Kustomize. Since we can commit any file to Git using Terraform, we can template any file the way we like. Let’s say we want to have a Kustomize overlay, and we would like to template some variables from Terraform. Here’s how we’d do it:

We generate a Kustomize overlay by creating a standard Terraform object, in the format which Kustomize expects. Like with Helm, we need to ensure that we’ll create an appropriate PR whenever the contents change. In the same way, we will SHA-256 encode the JSON-encoded string.

As we already did with Helm, we’ll also YAML encode the Kustomize object, which will generate a PR in the same manner as before.

Once the PR has been created and merged we can reference the Kustomize overlay in the same manner as if it was done natively.

This is a very simple example, you can apply the exact same methodology to Kustomize patches (JSON or merge patches) - the sky's the limit! There are a whole lot of ways to get creative while using Terraform, but hopefully these tidbits can show you how this is useful in practice. 

Summary

As this brief demo shows, this could be considered an evolution of the Gitops Bridge pattern. Perhaps we could call it Gitops Bridge 2.0 or Gitops Bridge++? This concept is still a work in progress, although it has already seen success in production environments. This post was inspired by an actual use case of one of our customers. Stay tuned for part 2 of this series!

If you have any questions feel free to hit me up on the CNCF Slack or on the Akuity Community Discord (you’ll find me as Blake Pettersson there). Also, feel free to play around and fork the repo using the examples above.