This blog post is a summary of a webinar we held live in June 2022. You can find the recording of the full 40-minute session on YouTube.
In this post we will be exploring multiple methods of managing secrets in GitOps with Argo CD specifically, however, these methods apply to other GitOps tooling as well.
Secrets management refers to the tools and methods for managing digital authentication credentials (secrets), including passwords, keys, APIs, and tokens for use in applications, services, privileged accounts and other sensitive parts of the IT ecosystem.
Oftentimes security is overlooked and treated as an afterthought in software delivery. In my experience as a DevOps consultant I have seen countless examples where passwords and other crucial information is left unencrypted or stored in plain text. Make sure that you always develop software and infrastructure with a security first approach using the best practices of DevSecOps. Also, I want to encourage you to consider what your needs are and stay open- minded to emerging solutions.
When you first start implementing Argo CD you will realize that all the tools in the Argo Project family are extremely flexible and un-opinionated which means you need to figure out which solution fulfills your needs in the best way possible.
Some things to consider, when selecting the way you manage secrets are:
In this post we will be looking at:
If you don’t have time to research the solutions, then we suggest External Secrets because it covers most of the use cases. But first, let’s take a look at what’s available.
The concept behind Sealed Secrets is essentially composed of two parts using source code to store secrets and utilizing a decryption key in the cluster. First you encrypt the secret on your computer, then it is decrypted on the cluster by the controller. To encrypt the secret you use a tool called Kubeseal, which has access to the cluster to retrieve an encryption key.
There is a custom resource named
SealedSecret which contains encrypted information and the controller decrypts it and creates a Kubernetes Secret. The sealed secret controller watches for any sealed secrets in the cluster across all namespaces, so everytime you create a new sealed secret the controller will detect it.
When to use:
Argo CD Vault Plugin works by replacing placeholders in secret manifests you store in Git with actual secrets. When Argo CD synchronizes the repo with the cluster it runs a plugin which finds the placeholder and replaces it with the data retrieved from the configured Secret Manager. This tool currently has a limitation of only being able to integrate with one secrets manager at a time (AVP is able to connect to any secrets manager, but there is only one configured backend per Argo CD, so you can have only one configured backend in your environment).
When to use:
SOPS is like if Sealed Secrets and Argo CD Vault Plugin had a baby. This tool uses keystores instead of a secrets manager. It’s a very popular option because of its age - people started using it when there weren’t any secrets managers, only keystores.
It’s similar to sealed secrets because you need to encrypt secrets manually. Where it differs from sealed secrets is that it reads the decryption key from a key store. It decrypts secrets in a similar fashion by using a binary that Argo CD runs (SOPS binary).
When to use:
Vault Agent is a tool created by Hashicorp specifically for Vault.
This solution works in a completely different way from the previously mentioned ones. Whenever you run a pod in the cluster, the controller injects a sidecar into your pod with the vault agent, which connects to the vault server and mounts a volume in the file system. Then whenever your application needs to read secrets it should read them from that mounted volume. This might be a good thing, but there is a price to pay in terms of maintenance and scalability - if you had 1000 pods, you would also have 1000s of agents running and making connections.
This method doesn’t use Kubernetes secrets - Kubernetes won’t know anything about your secrets.
When using this method, your applications have a dependency on Vault. If Vault is offline, the Vault agent will return error messages, but also your applications won’t be able to start.
When to use:
Secrets Store CSI Driver (Container Storage Interface) is a
DaemonSet that runs on each node. This approach scales a bit better than the agent-per-pod approach in Vault Agent. However it works a bit differently.
CSI is a special type of volume. In this method you will need to install the CSI Driver, then describe the volume in your workload. Because it’s a storage driver, it doesn’t watch for secret updates; it only runs on volume creation (Pod start/restart).
The secret provider class is the configuration which tells the driver how to connect to all the secrets managers.
When to use:
External Secrets works by creating a secret for each external secret. In the Custom Resource you specify the
SecretStore which can be either namespace or cluster scoped. The
SecretStore has the configuration to connect to the secrets manager (this can be cluster-scoped or namespace-scoped).
The ExternalSecret has a reference to the secret store, and it points to the exact location of your secret within this store.
This method also supports immutable secrets, instead of updating a secret you can create a new version of a secret.
When to use:
Traditionally when you want to connect to any Cloud, you have some type of secret key or password to authenticate. How do you initially connect to your secrets manager?
To solve this problem each cloud provider introduced their own solution which is IAM (Identity Access Management).
This works if you have your secrets manager and Kubernetes cluster in the same cloud. But what if you don’t?
For this scenario there is also a solution usually called Workload Identity Federation, but it might have different naming with different cloud providers.
After understanding all of the options I’m sure you’ve come to a similar conclusion as we have. We strongly advise you to use the External Secrets method, because this method is Kubernetes-native, secrets management agnostic, and rapidly being adopted by the community which means it will have continued support.
Typically in a mature organization you may already have certain limitations, situations, or secrets managers in use which means the solution that best fits your need may already be limited to one or two options that we shared. However, depending on your preferences regarding maintenance and scale, you may consider implementing a completely different solution.
When using Argo CD UI, you might want to match its design with your company's branding or wish to differentiate between the instances by giving different color…...
This year North America's "motor city", Detroit, hosted KubeCon. It wasn’t our first KubeCon as individuals, but it was our first KubeCon as a team. Our First…...