The Rendered Manifests Pattern

GitOps principles exist to address the genuine problems of visibility and collaboration when working with a complex system like Kubernetes. They stress the importance of a declarative desired state and continuous reconciliation. However, they leave considerable room for interpretation. We often get questions about how Git repos should be structured, what config management tools to use, and how branches should or should not be incorporated.

The configuration management tools you use won't significantly impact your ability to implement GitOps. Helm and Kustomize both address the need for a standardized set of manifests and the ability to alter them based on the specific environment they will be deployed into. This abstraction is extremely convenient for keeping your manifests DRY.

Yet, these abstractions create a new problem. They are typically referenced directly by the GitOps tooling to determine the desired state. A change to a Helm chart or a Kustomization base is a change to the abstraction; the true impact on the manifests deployed into environments is unclear.

Core Premise

The desired state for a Kubernetes cluster is not a Helm chart or a Kustomization; it's the manifests rendered by these config management tools. Sadly, this is typically what most organizations will store in Git when practicing GitOps. Any abstraction between git and kubectl can have undesired effects; you shouldn't mutate your source of truth right before it gets applied.

The desired state, stored and approved in Git, should contain no abstractions from what will be applied during reconciliation. It should be treated similarly to a container image, where it's immutable and applied as-is to the cluster. Having your GitOps tooling run Helm or Kustomize when applying the manifests to the cluster is like your container running apt-get install on start-up.

The solution to this is known as the Rendered Manifests pattern. Each change to the trunk (i.e. the main branch) of your GitOps repo will result in the manifests getting rendered by a CI workflow and stored as-is in Git. This artifact represents the cluster's desired state without any obfuscation.

The rendered manifests should be separated into environment-specific branches. As changes are made to these branches, the diff in the manifests between commits will be completely transparent. You'll be able to see plainly the effect of a change in main on each environment.

At a glance, the advantages of the Rendered Manifests pattern are:

• Eliminate obfuscation introduced by config management tooling (e.g. Helm, Kustomize) to improve visibility into the desired state.
• Reduce risk caused by built-in tooling with a truly immutable desired state.
• Improve performance for Argo CD by rendering manifests once.
• Set deployment and protection policies based on the environment.

There are notably two disadvantages:

• Shifting manifest rendering to the CI engine adds complexity.
• It does not work well with tooling that renders plain-text secrets (e.g. Sealed Secrets).

Understandably, you may still be skeptical, so let's break it down.

Isn't That Gitflow?

Let's get this out of the way first. On the surface, the Rendered Manifests pattern might sound like Gitflow, but it's not. You can continue to use short-lived feature branches or trunk-based development in your GitOps repo.

The environment-specific branches are not used for promotion between environments – changes in one environment's branch will not be merged into another branch. Instead, the contents of these branches are maintained by an automated workflow and are generated based on what's on the main branch.

Consider the contents of these branches akin to a release bundle containing the plain (rendered) manifests with the desired state of an environment. Similar to how a contributor should never pull a container image, modify the contents, and push it back to the registry, no one commits directly to the environment-specific branches. They are an artifact generated based on the contents of the main branch.

Eliminate Obfuscation to Improve Visibility

Take a look at the following example of a change to a Helm Umbrella chart maintained in a GitOps repo:

Looking at this diff, it's not intuitive what the result will be. You only know that the version of the Helm chart dependency is changing. It has no context of the changes to manifests rendered and applied to the cluster.

Look at another example using Kustomize:

To understand what will change, you'd need to check every environment that uses this Kustomize base.

If you want to be sure of the resulting manifests (the actual resources being deployed into the cluster), you need to run helm template or kustomize build. In Argo CD, the repo server performs this function. Each time the source of an Application changes or the cache expires, it runs helm template, kustomize build, or any custom tooling using a CMP to generate the manifests. Then they are passed to the application controller where it runs kubectl apply.

Changes to manifest should be continuously integrated and produce immutable artifacts, similar to code. Moving the manifest generation to your CI workflow and storing them on environment-specific branches will provide an immutable desired state where changes are clearly visible.

Using the Helm chart example from above, this is what the diff looks like when using the Rendered Manifests pattern. The one line change to the chart version resulted in almost a 1,000 lines changing in the resulting manifests.

Reduce Tooling Risk with an Immutable Desired State

An upgrade to the GitOps tooling can result in an upgrade to the baked-in toolchain (i.e. Kustomize, Helm), which risks affecting every environment it manages. Remember - any abstraction between git and kubectl can have undesired effects.

When the diff for upgrading Argo CD looks like this…

…how can you be sure what the effect on the manifests rendered for every Application in this Argo CD instance will be? (In this example, the upgrade from 5.42.0 to 5.43.0 of the argo-cd Helm chart upgrades Argo CD to v2.8.0 which includes upgrade helm version to 3.12.0.)

The core premise of continuous integration is fast feedback, catching errors before they make it into a live system. This same principle applies to the manifests that describe the desired state for your Kubernetes clusters. When your GitOps tooling is responsible for generating the manifests, issues won't be caught until after the changes have been approved, merged into main, and attempted to be applied to the cluster. At this point, fixing it requires repeating the entire process.

By moving manifest generation to your CI workflow and storing the manifests in environment-specific branches, you achieve guaranteed stability for your cluster's desired state. The raw YAML is captured in an immutable version (a commit). Argo CD no longer transforms what's in Git; it applies it directly to the cluster without modification.

Improve Performance for Argo CD

Running Kustomize and Helm in Argo CD is expensive. Each time the target revision for an Application changes, the cache expires, or someone performs a hard refresh, the repo server needs to re-generate the manifests.

It's easy to see how expensive the manifest generation is when you look at the resource requests for the repo server in large Argo CD deployments. I've seen cases where the repo server is given 32 CPUs and 200GiB of memory!

This is especially the case for mono-repos, where a change to the manifests in one folder can unnecessarily result in Argo CD regenerating the manifests for many Applications.

By leveraging the CI engine to generate and store the raw manifests on environment-specific branches, it only happens once using on-demand compute (as opposed to continually reserving resources for the repo server).

Automate Dev and Protect Production

So far, the advantages of this pattern can mainly be attributed to the “you should render manifests in CI” part. The second part, “and store them in environment-specific branches,” now comes into play.

Separating the rendered manifests into branches gives you the added benefit of setting different policies for each environment.

Consider again the diff showing a change to a Kustomize base.

Without the rendered manifests pattern and using an automated sync policy, this change would be applied to every environment simultaneously.

Using the Rendered Manifests pattern, changing the kustomize base on the main branch will result in the manifests being rendered in CI. Likely, the manifests for your non-production environments can be deployed automatically, so they get pushed directly to the environment-specific branches.

Most organizations are uncomfortable doing the same for production environments (very few are mature enough to practice continuous deployment). Instead, the manifests rendered for the production environment can be pushed to a short-lived branch. Then, a PR is opened, proposing the change to the environment-specific branch containing the existing desired state.

Because the manifests are already rendered, the diff shown in the PR will represent the changes without obfuscation or abstractions. Those reviewing and approving the PR can easily understand the true impact of the change.

Most Source Code Management (SCM) providers have extensive fine-grained permission control for branches. Using these, you can ensure that any change to the desired state for production has followed the correct processes. For example, a code owner has given an approving review, any desired checks have run successfully, and no changes are ever pushed directly to the production branch.

You might think, ”Well, I can get a similar solution in Argo CD by using manual syncs on my production Applications.” While this is true, the disadvantage is that the desired state is ambiguous while your Applications sit in an out-of-sync state. Argo CD shows the diff, but the generated manifests could vary when the sync is performed.

Disadvantages

Added CI Complexity

There's no denying that this pattern adds additional CI automation requirements. The simplicity of using Argo CD for manifest generation should not be understated. Much work has been done in Argo CD to integrate Helm and Kustomize and provide reliable manifest generation.

For Argo CD, there have been several attempts to add diffs of the rendered manifests from Applications to pull requests. I created a GitHub Action called argocd-diff-action to solve this problem (it currently sits broken, as I opted to use the rendered manifests pattern to produce immutable artifacts with clean and informative diffs to the desired state of my Kubernetes clusters). Zapier recently open-sourced their in-house tool kubechecks, which performs a similar function with added features like policy checks.

At Akuity, we created a tool used internally to adopt the rendered manifests pattern. While the project is still under active development, the goal is to open-source it so that Argo CD users can take their existing definitions of what manifests to render (i.e. an Application manifest) and use the tool to render them in CI instead of with Argo CD.

Rendered Plain-text Secrets

Kubernetes Secrets management tools like Kustomize + SOPS are incompatible with the Rendered Manifests pattern. They allow users to store encrypted secrets in Git and rely on the tooling running in the cluster to render the manifests and decrypt the secrets.

This is not ideal for the rendered manifest pattern, where the decrypted secrets would end up in plain text on the environment-specific branches.

Before adopting this pattern, it's recommended to use a tool like External Secrets Operator, which uses ExternalSecret resources that contain a reference to data in a SecretStore and are safe to store in Git. It then uses an in-cluster controller to generate the Kubernetes Secret based on the ExternalSecrets. In our blog post,_ How to manage Kubernetes secrets with GitOps?_, we explain why this is our preferred method, regardless of whether you use the Rendered Manifests pattern.

Conclusion

While GitOps principles emphasize declarative desired states and continuous reconciliation, they leave much room for interpretation when structuring Git repositories, selecting configuration management tools, and managing branches. The Rendered Manifests pattern rectifies these issues by ensuring that the desired state stored in Git remains unobscured and immutable, and it is applied to the cluster without transformation or abstraction.

This pattern improves visibility, eliminates obfuscation, and reduces tooling risks, leading to better security and safer change management. It enhances performance, especially in large Argo CD deployments, by shifting the manifest generation process to CI workflows. Along with the ability to set different policies for each environment, adding flexibility and control to your GitOps workflow while protecting production environments.

If you want to learn more about the Rendered Manifests pattern, check out our Advanced GitOps workshop. Be on the lookout for a free in-person workshop near you! If you have made it this far and are still not convinced, contact me on LinkedIn and tell me why!