GitOps Best Practices: A Complete Guide for Modern Deployments

GitOps is quickly becoming the de facto standard in how to operate a cloud-native ecosystem using Kubernetes as the core orchestration layer. Kubernetes changed the game. So much so that conventional Infrastructure as Code (IaC) tooling fell short and management tools such as Argo CD and Flux emerged to leverage the declarative nature of Kubernetes. In this blog, we will go over the lessons learned from running Argo CD in production and at scale, and share some valuable insights along the way.

As the creators of Argo CD, long-time GitOps practitioners, and Kubernetes experts, we wanted to share GitOps best practices, insights, and lessons learned from deploying Argo CD at scale in production.

Want to see GitOps Best Practices in action? Watch this video on how Argo CD enables GitOps at scale.

What is GitOps? 

Argo CD was built from the ground up with GitOps in mind. GitOps, which was coined by Alexis Richardson, founder and CEO of WeaveWorks, has been a topic that has been discussed for a while. GitOs initially started as a way to describe how you use Git as the source of truth for your infrastructure configurations. It later evolved into becoming an operational framework for managing application deployments using a declarative approach. This paved the way for what we have now, the GitOps Principles.

The Open GitOps Principles

Today, GitOps has established itself as a foundation of cloud-native technologies and attracted a vendor-neutral group that developed the Open GitOps Principles.

A system managed by GitOps is:

• Declarative: A system managed by GitOps must have its desired state expressed declaratively.
  
  

• Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
  
  

• Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
  
  

• Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.

The GitOps principles do a great job to define the industry standard. But, these principles do not tell you how to implement GitOps, or even give you best practices. Instead, they are meant to guide you in your implementation. While defining the end goal of GitOps is a good first step, there is a need to go a step further and share best practices from the team that built the Argo Project.

Manifest Templating and Patching

A lot of folks run into the same issue the first time they start out with GitOps: “I’m dealing with a LOT of YAML!”. The issue of duplication and problems against the “DRY” principle seem to come to a head rather quickly. 

It all really comes down to this: you’ll have to duplicate a lot of the same YAML after you consider things like environments, clusters, regulatory restrictions, and anything else in your organization that might force you to create a lot of YAML with only slight variations between files. Luckily this has been solved before and you can take advantage of built-in Kubernetes-native tooling.

Kustomize

If you’ve been working with Kubernetes, using Kustomize shouldn’t be something new to you. To those of you who are new to it, Kustomize is a patching framework that allows you to take Kubernetes manifests and overlay changes to them - leaving the original manifest intact. Instead, Kustomize renders a new manifest with the resulting changes. Let’s take a simple example of changing the replicas of a deployment from 1 to 3.

First, a high level overview of the directory structure layout for Kustomize:

$ tree
.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays
    └── dev
        └── kustomization.yaml

The top level directories are base and overlays with another directory under overlays called dev. It doesn’t necessarily “matter” what these directories are named or how they are structured per se, but these are de facto standards which is fine for the purposes of this paper.

The deployment.yaml file is a basic Kubernetes Deployment manifest. However, there is another manifest in the base directory: the kustomization.yaml file. Let’s take a look at this file.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml

In this specific instance, nothing is necessarily “special” about this file. You can deduce by looking at the contents of this file that you’re instructing Kustomize to “read in” the deployment.yaml file - and you’d be right! But the real magic exists in the overlays/dev directory! Let’s take a look at that kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
  - target:
      name: nginx
      group: apps
      kind: Deployment
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
  - target:
      name: nginx
      group: apps
      kind: Deployment
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
  - target:
      name: nginx
      group: apps
      kind: Deployment
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3

Here, you can see that I am telling Kustomize to: 1) read in any resources that are provided by the Kustomization file in the ../../base directory, and 2) replace the replicas on the named Deployment to be 3. You run the following command to render the YAML.

$ kustomize build overlays/dev

$ kustomize build overlays/dev

$ kustomize build overlays/dev

Also, you could have run kubectl apply -k overlays/dev/ to apply the manifest directly onto a Kubernetes cluster.

The output of the build command should have rendered a new Deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}

So how does this help? Stepping back and looking at your application deployments from environment to environment - there isn’t a big difference between applications running in different environments. But, while functionally the differences are not big, they are substantial.

For example; database credentials, scale, container images, etc. wholly change how an application behaves. However, the declarations stay structurally the same and only the values of these are changing. This makes Kustomize a powerful tool to use to deploy the same application to different environments with only the variants between them being stored.

Helm

If you’ve worked on Linux systems in the past, you may be familiar with tools like apt or dnf/yum. These tools were meant to help you install, manage, and maintain the lifecycle of applications you were installing on a system. Similarly, Helm had the same goal in mind. Now with Helm v3, the popularity of the go-to package manager for Kubernetes has grown!

At its core, Helm can be seen as a “templating engine” for Kubernetes application deployments. Helm takes a created template where an end user can supply values and it will generate a payload that a user can version, upgrade, or modify using the Helm framework. This diagram shows how it works.

Helm consists of charts, which are packaged and templatized versions of your YAML manifests. You can inject values into the parameters defined in the templates, and Helm injects these values into the manifests to create a release. A release is the endstate representation of the YAML that is deployed to your Kubernetes cluster. The information is stored as a secret on the Kubernetes cluster. Helm has a large ecosystem and many Helm repositories are available for end users to use to deploy pre-built applications and many ISVs (Independent Software Vendors) use Helm as a deployment mechanism for their software.

Helm provides not only a method of packaging an application and parameterizing YAML manifests but also a templating engine that can deploy your application to different environments. In this way, you can use Helm to bundle your applications and “rubber stamp” deploy your applications with only needing to tweak the values needed for each deployment. 

Here’s an example of adding a repo and deploying a Helm release.

$ helm repo add akuity-demos https://akuity.github.io/demo-helm-charts/
"akuity-demos" has been added to your repositories

$ helm install myapp --create-namespace --namespace example --set replicaCount=3 akuity-demos/simple-go
NAME: myapp
LAST DEPLOYED: Mon Sep 18 16:16:53 2023
NAMESPACE: example
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Simple Go App deployed!

$ kubectl get pods -n example
NAME                           	READY   STATUS	RESTARTS   AGE
myapp-simple-go-64c587f7b4-6htcs   1/1 	Running   0      	8s
myapp-simple-go-64c587f7b4-f2bmc   1/1 	Running   0      	8s
myapp-simple-go-64c587f7b4-r6tdg   1/1 	Running   0      	8s

As you can see, using Helm, you’ll only need to store the Values file for each deployment of an application.

Which one to use?

If you are using mainly raw Kubernetes manifest files, your best bet is to use Kustomize as much as possible. It’s not only built into Kubernetes, but it’s built into many GitOps tools as well.

If you need to parameterize certain things or come from the Helm world, you might think about either leaning all into Helm or starting to write Helm charts. Helm is valuable when you want to parameterize your configurations instead of patching them. Helm is also great when you’re consuming third party application stacks provided by ISVs. You usually parameterize when you don’t know something about the cluster ahead of time (for example, the Ingress “host” field in the YAML). You can use Helm to parameterize the configuration and just supply the values specific for that deployment.

However, It’s not a question of “Kustomize versus Helm,” but rather “Kustomize and Helm.” Most of the time, you will use them in tandem. Take a look at the Akuity Blog post about Helm and how you can use it with Kustomize for more info.

Git Workflows: Best Practices for Managing Repositories

Git workflows have been the centerpiece of application development for a long time and have become the standard in application development and deployment. With the adoption of GitOps and the popularity of Infrastructure as Code, your Git not only becomes your source of truth, but also your interface to how you manage your environment. Git workflows are familiar with these workflows and now operation teams are also using similar workflows.

Instinctively, a lot of organizations want to adopt git-flow, since it has been the de facto process for so long. But there are key differences between how you manage your code vs how you manage your GitOps repository.

Separate Workflows for Code vs. Deployments

The first challenge that organizations face is what to do with the code that runs your application vs the manifests that deploy them? The answer is pretty straightforward: separate them.

While organizations are using Git flow for their application development, many DevOps engineers are adopting trunk based development for their GitOps repositories. These are two fundamentally different types of workflows and can cause issues when an update like changing the replica count of a deployment (which means that the code is the same) requires a rebuilding and retesting of a codebase that hasn’t changed (and likely already in production). Also, the approval process for an environment change differs from a code change and shouldn’t hinder the continuous integration process for developers.

No Branches for Environments

Another paradigm shift that most organizations need to make is the idea of keeping environment specific configurations in separate folders and not in branches (see the section on kustomize for an example of how this looks). This practice might seem odd in the beginning, but in practice, makes life managing these environments easier.

From a high level, you don’t use long-lived branches for environments because a promotion isn’t as simple as a merge from one branch to another. Since you’re using Git, your instinct is to treat your Kubernetes YAML like code (it’s even in the name: infrastructure as code). However, you need to keep in mind that you’re promoting manifests, not code. Environment specific configuration shouldn’t be merged, for example, Secrets and ConfigMaps are fundamentally different between environments. Promoting an updated image can be a nightmare with Git Flow since you have to cherry pick every change; which may cause overhead and is more trouble than it’s worth.

In short, using Trunk based development together with Kustomize and Helm to help templatize a lot of similarities (while leaving room for customization) is an ideal way to simplify your GitOps workflows.

GitOps Repository Directory Structures

A lot has been written about GitOps Repository structures, since it’s usually one of the first challenges that many organizations run into when implementing GitOps. Unfortunately, there is no magic bullet when it comes to the “best” or “preferred” repository layout. The issue has mainly to do with Conway’s Law, which states:

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure." - Melvin E. Conway

Meaning, that how your organization is structured will dictate how your directory structure (and your workflow in general) is formatted (and not the other way around). Organizational boundaries and separation of responsibilities will also have a great influence on how your directories are structured.

So the practice of what a good structure looks like will differ from organization to organization, however, there are some high-level best practices and principles that you can follow. These can guide your organization to the optimal directory structure that works best for you. We’ve gone through many of these already (which will be highlighted again for the sake of completeness) and I’ll introduce others that can help guide you.

Keeping it DRY

Similar to programming principles, you shouldn’t repeat yourself and should apply the practice of DRY - “Don’t Repeat Yourself”. We can augment this to “Don’t Repeat YAML”. The idea being, as described in the Manifest Templating and Patching section, storing everything in Git can lead to a lot of the same YAML. Using the strategies described in that section (of using Kustomize and Helm) you can avoid repeating a lot of the same YAML over and over again. This will keep your repository clean and easy to understand. Specifically, use Kustomize to keep the base configuration of your deployment and then store the deltas as patched overlays when needed.

Parameterize

There are certain situations where patching with Kustomize isn’t feasible at all. Yes, patching YAML is easy when you already know the values beforehand. However, there are situations where you won’t know the value of something until it gets applied to the destination cluster. A classic example of this is an Ingress Object. This configuration has a host field in the YAML manifest that is supposed to be filled in with the fully qualified domain name (FQDN) of the application being deployed. When you are deploying onto many clusters, the FQDNs of each one may not be known beforehand. Parameterizing your configurations makes sense in this scenario. This is where Helm shines, specifically when you use the lookup feature.

In the end you’ll use a combination of Kustomize and Helm. Using both, you should be able to limit YAML duplication in your GitOps repository.

Directory Structure Examples

As I mentioned in the beginning of this section, your Git repository structure will depend heavily on how your organization is laid out. The repositories reflect how your organization communicates with each other and how your current deployment workflow is represented. The different organizations with different workflows are often referred to as silos, but it’s more accurate to call them “boundaries”. Just as developers won’t modify platform configurations, operators who work on platforms normally won’t go in to change developers’ source code.

Since it’s very difficult to outline one “true” way; I’ve collected some examples that can help you get on your way:

• Christian Hernandez’s (Akuity) GitOps Example Repo: This repo outlines how a 1:1 Repo to Cluster layout looks like. This is using Argo CD’s ApplicationSets for the bootstrapping.

• Gerald Nunn’s (Red Hat) GitOps Standards: This repo goes over Gerald’s GitOps standards used in his work. It also goes over an example repository layout that includes how to handle multiple clusters from a Monorepo.

• Johannes Schnatterer (Cloudogu GmbH) GitOps Repository Structures and Patterns: This blog goes over some of the pros and cons of doing an “Repo per team” vs “Repo for Application”

• Terraform'ed Argo CD: Terraform is a popular tool for creating the underlying infrastructure for cloud-native applications. This includes Kubernetes clusters, databases, Identity and Access Management roles, etc (aka “cloudy stuff”). However, it is challenging to integrate the outputs of Terraform resources to be used with Argo CD.. 

Take Your GitOps Workflow to the Next Level

This blog explored key best practices for implementing GitOps workflows effectively. While every organization may have unique challenges, these guidelines serve as a foundational framework to navigate the complexities of GitOps adoption. By following these best practices, teams can streamline deployments, enhance security, and improve operational efficiency.

For a deeper dive into GitOps best practices, including rendered manifests, CI/CD integration, continuous promotion strategies, and more download the whitepaper and find out how to implement GitOps and what are the best practices for running Argo CD in production.