Enforcing Kubernetes Best Practices with Kyverno and Argo CD
Nicholas Morey
If you've landed on this blog post, I assume you're already familiar with Kubernetes and know it's a container orchestration platform. As new application releases are created, you've got your CI processes down for building container images. Kubernetes is fantastic in providing a wide range of functionality that enables users to perform complex deployment strategies. The challenge is that there are good and bad ways to use it, depending on your environment.
The platform engineers in your organization are likely intimately familiar with Kubernetes and understand the nuances of the right and the wrong way to deploy something into the cluster based on the specifics of your organization. The challenge comes from disseminating this information to all of the users of the platform.
Most of the operators' understanding likely came from formal training or spending many hours building the platform and learning from their mistakes. It would be unrealistic to require that all of the application developers that intend to interact with the platform go through the same experience. It's expensive for the organization in terms of time, effort, and potential impact on the product and the customers as these lessons are learned.
Inside every organization, there are policies and standards that users of the internal development platform are aware of or are required to follow. The challenge is that many organizations use documentation and broad communication to ensure users follow these standards. This provides a long feedback loop between people deviating from the expected standards and learning the correct method.
Kyverno
Enter Kyverno, a Kubernetes-native policy engine that provides a way to codify the lessons learned by platform administrators. Kyverno deploys directly into a Kubernetes cluster and uses custom resources for policy definitions. Using admission webhooks, Kyverno can audit policy violations or go as far as blocking requests as they attempt to enter the cluster.
A policy engine can significantly reduce the feedback loop between attempting to apply a change in Kubernetes and learning that deployment does not meet the standards set by the platform's administrators. As application developers interact with the cluster, the policies provide feedback immediately on if they are deviating from the best practices set out by the platform team.
For example, a developer becoming familiar with Kubernetes decides to deploy the latest tag for their application. This may seem harmless, but they are unaware of the dangers of using a variable tag. Until this mistake is caught by a cluster administrator and reported back to the user, a lot of damage can be done in that time.
Instead, when the user goes to deploy the latest tag into the cluster, they can be immediately prompted with a clear message explaining why this is not allowed and steps to remediate the policy violation. This enables them to continue with what they were trying to accomplish, deploying their application into the cluster.
Organizations can even integrate policies into the CI/CD workflow to catch any violations earlier in the release cycle. If somehow a violation makes it to the cluster, Kyverno can block the admission request to enforce the policy or even mutate resources as they are applied so that they conform with best practices.
GitOps / Argo CD
As a Kubernetes administrator, you're probably curious about the best way to manage these policies; this is where Argo CD and GitOps come in.
GitOps brings best practices for application development, such as version control, collaboration, compliance, and tooling, and applies them to Kubernetes. The manifests for Kubernetes resources used in a cluster are stored in Git, which provides an immutable version of the desired state.
Argo CD acts as the GitOps agent, which continuously reconciles the difference between the desired state and the live state of the cluster. It takes any changes in Git and applies them to the Kubernetes cluster.
Like any Kubernetes manifest, Kyverno policies can be stored and managed through Git. This enables them to follow transparent approval processes and provide an audit trail of when policies change and why.
Kyverno Policies for Best Practices
Let us look at a typical example of enforcing Kubernetes best practices with Kyverno policies. We'll start by adding a policy to the cluster using Argo CD. For this example, Kyverno and Argo CD are already installed into the cluster.
In the GitOps repository, add the require-resource-requests policy, which requires that pods have defined resource requests. This is critical to Kubernetes' ability to schedule pods onto nodes with sufficient resources. Without them, in terms of resource availability, Kubernetes will schedule pods blindly onto nodes.
The validationFailureAction field determines the behavior of how Kyverno responds to a failed validation check. The require-resource-requests policy is set to enforce a rule failure, blocking any request that violates them. When this happens, it will record a fail (blocked) event on the policy resource responsible.
The validate-resource-requests rule will match any Pod resource in the dev namespace, even if generated from a higher-level resource, such as a Deployment or ReplicaSet. It will validate that the cpu and memory resource request fields on the containers in the pod have a value defined. When the policy validation fails, it will include the message defined in the rule.
We'll add an Application to Argo CD that points to the directory in the repo containing the policy.
With the policies synced into the cluster, we will deploy another Argo CD Application containing two nginx pods, one that will pass policy validation and the other that will fail due to missing resource requests.
The Sync will ultimately fail because the policy is set to block any resources that contain a violation from being created in the cluster. Looking at the Sync Status in Argo CD, the user can see in the message which Kyverno policy is failing and the message from the rule explaining why.
The Sync Status will also break down the result for each resource, making it clear to the user which resource failed and why.
On the policy resource, the events will display the violation as well.
Once the policy violation is fixed for the resource, the Argo CD Application will then be able to sync the resource into the cluster.
Conclusion
Kyverno, as a policy engine, is a powerful tool for providing clear feedback to the users of an internal developer platform on achieving the best practices set out by the Platform team. However, this is only the tip of what Kyverno is capable of. The example in this post demonstrates the admission control functionality, but policies can also mutate, generate, and clean up resources. This creates the option to enforce policies by automatically updating resources as they enter the cluster or building a self-service platform that responds to resources created in the cluster to create new clusters and bootstrap them using Argo CD.
Managing the policies for a cluster is simplified when adhering to GitOps principles and using Argo CD to manage the deployment. The Argo CD UI provides a clear understanding of what resources are blocked by policy validations, making it easy for users to understand how to bring the resources into compliance.
Together, Kyverno and Argo CD open up a whole new world of cluster administration that can scale with your organization's needs. For more best practices in using Argo CD and Kyverno head to "3 Essential Tips for Using Argo CD and Kyverno" on Nirmata's blog.
Get in Touch
Are you using Kyverno and Argo CD together? Check out these Argo CD policies. You can find the Kyverno community the Kubernetes Slack.
Additional Resources
Loved this blog post? Want to learn more about helpful tools you can utilize with Argo CD? Check out these blog posts:
[Blog Post]: Argo CD CDK8S Config Management Plugin
[Blog Post]: Akuity Crossplane Support
[Blog Post] Argo Security Automation with OSS-Fuzz
[Blog Post]: Yet Another Take on Integrating Terraform with Argo CD
Frequently Asked Questions About Kyverno and Argo CD
How do I store Kyverno policies for Argo CD deployments?
Kyverno policies should be treated like Kubernetes manifests and stored in a Git repository alongside your application manifests. Argo CD can then deploy and manage these policies declaratively, ensuring version control, traceability, and auditability. This approach also aligns with GitOps principles, keeping policy changes transparent and reversible.
What’s the best approach to start with validation before mutation in Kyverno policies?
Begin by implementing ValidatingPolicy or ClusterPolicy to enforce standards and audit non-compliant resources without changing them. Once teams are familiar with the rules and compliance is established, you can gradually introduce mutation policies to automatically correct or enforce desired states. This staged approach minimizes disruption and builds confidence in automated enforcement.
How do I prevent reconciliation loops caused by Kyverno’s mutating webhook in Argo CD?
Kyverno’s mutating webhook can modify resources during policy application, causing Argo CD to continuously detect differences between the desired and actual states. To prevent reconciliation loops, configure Argo CD to ignore specific fields or adjust synchronization settings. Proper readiness/liveness probes and health checks also help ensure that resources are only considered “healthy” when fully compliant.
How can I enforce image security with Kyverno policies in a GitOps workflow?
Use ImageValidatingPolicy to verify image signatures, validate SBOMs and attestations, and restrict allowed registries. This ensures that only trusted and compliant container images are deployed to your cluster. Combined with immutable tags, these policies reduce supply chain risks and improve overall Kubernetes security.
