How to manage Kubernetes secrets with GitOps?

This blog post is a summary of a webinar we held live in June 2022. You can find the recording of the full 40-minute session on YouTube.

In this post we will be exploring multiple methods of managing secrets in GitOps with Argo CD specifically, however, these methods apply to other GitOps tooling as well.

Secrets management refers to the tools and methods for managing digital authentication credentials (secrets), including passwords, keys, APIs, and tokens for use in applications, services, privileged accounts and other sensitive parts of the IT ecosystem.

Oftentimes security is overlooked and treated as an afterthought in software delivery. In my experience as a DevOps consultant I have seen countless examples where passwords and other crucial information is left unencrypted or stored in plain text. Make sure that you always develop software and infrastructure with a security first approach using the best practices of DevSecOps. Also, I want to encourage you to consider what your needs are and stay open- minded to emerging solutions.

When you first start implementing Argo CD you will realize that all the tools in the Argo Project family are extremely flexible and un-opinionated which means you need to figure out which solution fulfills your needs in the best way possible.

Some things to consider, when selecting the way you manage secrets are:

• Scale and scalability
  • If you expect your organization to scale, it may make sense to put a bit more effort into a solution that is complex to configure in the beginning instead of the easiest solution to implement.
• How to maintain the selected solution
  • What is the developer experience to rotate secrets? Or to maintain sidecars?
• Your particular use case
  • You might need to implement a solution quickly to meet a deadline, or you may have a good reason to use a specific solution like a pre-existing architecture.

What solutions for managing secrets are out there?

In this post we will be looking at:

• Sealed Secrets
• Argo CD Vault Plugin
• SOPS (Secrets OPerationS)
• Vault Agent
• Secrets Store CSI Driver
• External Secrets

If you don’t have time to research the solutions, then we suggest External Secrets because it covers most of the use cases. But first, let’s take a look at what’s available.

Sealed Secrets

The concept behind Sealed Secrets is essentially composed of two parts using source code to store secrets and utilizing a decryption key in the cluster. First you encrypt the secret on your computer, then it is decrypted on the cluster by the controller. To encrypt the secret you use a tool called Kubeseal, which has access to the cluster to retrieve an encryption key.

There is a custom resource named SealedSecret which contains encrypted information and the controller decrypts it and creates a Kubernetes Secret. The sealed secret controller watches for any sealed secrets in the cluster across all namespaces, so everytime you create a new sealed secret the controller will detect it.

Features:

• Sealed Secrets are stored in Git, which means this method doesn’t need a connection to any Secrets Manager.
• You have to manually encrypt each secret.

When to use:

• When you have a small number of clusters or as a temporary solution for a quick start.

Argo CD Vault Plugin

Argo CD Vault Plugin works by replacing placeholders in secret manifests you store in Git with actual secrets. When Argo CD synchronizes the repo with the cluster it runs a plugin which finds the placeholder and replaces it with the data retrieved from the configured Secret Manager. This tool currently has a limitation of only being able to integrate with one secrets manager at a time (AVP is able to connect to any secrets manager, but there is only one configured backend per Argo CD, so you can have only one configured backend in your environment).

Features:

• It doesn’t require Custom Resources and controllers.
• It supports a wide variety of secrets managers.
• It can only connect to one secrets manager at a time.
• Requires a hard refresh when a secret is updated.

When to use:

• When you don’t want CRDs and controllers.
• When you only manage resources with Argo CD.

SOPS (Secrets OPerationS)

SOPS is like if Sealed Secrets and Argo CD Vault Plugin had a baby. This tool uses keystores instead of a secrets manager. It’s a very popular option because of its age - people started using it when there weren’t any secrets managers, only keystores.

It’s similar to sealed secrets because you need to encrypt secrets manually. Where it differs from sealed secrets is that it reads the decryption key from a key store. It decrypts secrets in a similar fashion by using a binary that Argo CD runs (SOPS binary).

Features:

• Doesn’t require Custom Resources and controllers.
• Stored in Git.
• Manual encryption of secrets.
• Needs SOPS plugin for Argo CD.

When to use:

• In 2018

Vault Agent

Vault Agent is a tool created by Hashicorp specifically for Vault.

This solution works in a completely different way from the previously mentioned ones. Whenever you run a pod in the cluster, the controller injects a sidecar into your pod with the vault agent, which connects to the vault server and mounts a volume in the file system. Then whenever your application needs to read secrets it should read them from that mounted volume. This might be a good thing, but there is a price to pay in terms of maintenance and scalability - if you had 1000 pods, you would also have 1000s of agents running and making connections.

This method doesn’t use Kubernetes secrets - Kubernetes won’t know anything about your secrets.

When using this method, your applications have a dependency on Vault. If Vault is offline, the Vault agent will return error messages, but also your applications won’t be able to start.

Features:

• Doesn’t create a Kubernetes secret
• Uses templates for rendering files.
• Can update mounted files.
• Only works with Hashicorp Vault.

When to use:

• You only use Vault.
• You don’t want Kubernetes secrets.

Secrets Store CSI Driver

Secrets Store CSI Driver (Container Storage Interface) is a DaemonSet that runs on each node. This approach scales a bit better than the agent-per-pod approach in Vault Agent. However it works a bit differently.

CSI is a special type of volume. In this method you will need to install the CSI Driver, then describe the volume in your workload. Because it’s a storage driver, it doesn’t watch for secret updates; it only runs on volume creation (Pod start/restart).

The secret provider class is the configuration which tells the driver how to connect to all the secrets managers.

Features:

• Very flexible option. It doesn’t create a Kubernetes secret by default, but can.
• Can’t update mounted files
• DaemonSet

When to use:

• You have multiple different types of secrets managers.
• You don’t want Kubernetes secrets.

External Secrets

External Secrets works by creating a secret for each external secret. In the Custom Resource you specify the SecretStore which can be either namespace or cluster scoped. The SecretStore has the configuration to connect to the secrets manager (this can be cluster-scoped or namespace-scoped).

The ExternalSecret has a reference to the secret store, and it points to the exact location of your secret within this store.

This method also supports immutable secrets, instead of updating a secret you can create a new version of a secret.

Features:

• Connects to multiple sources, including other Kubernetes clusters.
• Can use templates.
• Updates secrets discreetly.

When to use:

• When you might have multiple different secrets managers.
• You want to use Kubernetes secrets.

Secrets Management and the Cloud

Traditionally when you want to connect to any Cloud, you have some type of secret key or password to authenticate. How do you initially connect to your secrets manager?

To solve this problem each cloud provider introduced their own solution which is IAM (Identity Access Management).

This works if you have your secrets manager and Kubernetes cluster in the same cloud. But what if you don’t?

For this scenario there is also a solution usually called Workload Identity Federation, but it might have different naming with different cloud providers.

Conclusion

After understanding all of the options I’m sure you’ve come to a similar conclusion as we have. We strongly advise you to use the External Secrets method, because this method is Kubernetes-native, secrets management agnostic, and rapidly being adopted by the community which means it will have continued support.

Typically in a mature organization you may already have certain limitations, situations, or secrets managers in use which means the solution that best fits your need may already be limited to one or two options that we shared. However, depending on your preferences regarding maintenance and scale, you may consider implementing a completely different solution.