Enforcing Kubernetes Best Practices with Kyverno and Argo CD

If you've landed on this blog post, I assume you're already familiar with Kubernetes and know it's a container orchestration platform. As new application releases are created, you've got your CI processes down for building container images. Kubernetes is fantastic in providing a wide range of functionality that enables users to perform complex deployment strategies. The challenge is that there are good and bad ways to use it, depending on your environment.

The platform engineers in your organization are likely intimately familiar with Kubernetes and understand the nuances of the right and the wrong way to deploy something into the cluster based on the specifics of your organization. The challenge comes from disseminating this information to all of the users of the platform.

Most of the operators' understanding likely came from formal training or spending many hours building the platform and learning from their mistakes. It would be unrealistic to require that all of the application developers that intend to interact with the platform go through the same experience. It's expensive for the organization in terms of time, effort, and potential impact on the product and the customers as these lessons are learned.

Inside every organization, there are policies and standards that users of the internal development platform are aware of or are required to follow. The challenge is that many organizations use documentation and broad communication to ensure users follow these standards. This provides a long feedback loop between people deviating from the expected standards and learning the correct method.

Kyverno

Enter Kyverno, a Kubernetes-native policy engine that provides a way to codify the lessons learned by platform administrators. Kyverno deploys directly into a Kubernetes cluster and uses custom resources for policy definitions. Using admission webhooks, Kyverno can audit policy violations or go as far as blocking requests as they attempt to enter the cluster.

A policy engine can significantly reduce the feedback loop between attempting to apply a change in Kubernetes and learning that deployment does not meet the standards set by the platform's administrators. As application developers interact with the cluster, the policies provide feedback immediately on if they are deviating from the best practices set out by the platform team.

For example, a developer becoming familiar with Kubernetes decides to deploy the latest tag for their application. This may seem harmless, but they are unaware of the dangers of using a variable tag. Until this mistake is caught by a cluster administrator and reported back to the user, a lot of damage can be done in that time.

Instead, when the user goes to deploy the latest tag into the cluster, they can be immediately prompted with a clear message explaining why this is not allowed and steps to remediate the policy violation. This enables them to continue with what they were trying to accomplish, deploying their application into the cluster.

Organizations can even integrate policies into the CI/CD workflow to catch any violations earlier in the release cycle. If somehow a violation makes it to the cluster, Kyverno can block the admission request to enforce the policy or even mutate resources as they are applied so that they conform with best practices.

GitOps / Argo CD

As a Kubernetes administrator, you're probably curious about the best way to manage these policies; this is where Argo CD and GitOps come in.

GitOps brings best practices for application development, such as version control, collaboration, compliance, and tooling, and applies them to Kubernetes. The manifests for Kubernetes resources used in a cluster are stored in Git, which provides an immutable version of the desired state.

Argo CD acts as the GitOps agent, which continuously reconciles the difference between the desired state and the live state of the cluster. It takes any changes in Git and applies them to the Kubernetes cluster.

Like any Kubernetes manifest, Kyverno policies can be stored and managed through Git. This enables them to follow transparent approval processes and provide an audit trail of when policies change and why.

Kyverno Policies for Best Practices

Let us look at a typical example of enforcing Kubernetes best practices with Kyverno policies. We'll start by adding a policy to the cluster using Argo CD. For this example, Kyverno and Argo CD are already installed into the cluster.

In the GitOps repository, add the require-resource-requests policy, which requires that pods have defined resource requests. This is critical to Kubernetes' ability to schedule pods onto nodes with sufficient resources. Without them, in terms of resource availability, Kubernetes will schedule pods blindly onto nodes.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-requests
  annotations:
    policies.kyverno.io/title: Require Resource Requests
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/description: >-
      As application workloads share cluster resources, it is important to specify resources
      requested and consumed by each Pod. Requiring resource requests per Pod is recommended,
      especially for memory and CPU. If a Namespace level request is specified, defaults will
      automatically be applied to each Pod based on the LimitRange configuration. This policy
      validates that all containers have something specified for memory and CPU requests.
spec:
  validationFailureAction: enforce
  background: true
  rules:
    - name: validate-resource-requests
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - dev
      validate:
        message: 'CPU and memory resource requests are required.'
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: '?*'
                    cpu: '?*'

The validationFailureAction field determines the behavior of how Kyverno responds to a failed validation check. The require-resource-requests policy is set to enforce a rule failure, blocking any request that violates them. When this happens, it will record a fail (blocked) event on the policy resource responsible.

The validate-resource-requests rule will match any Pod resource in the dev namespace, even if generated from a higher-level resource, such as a Deployment or ReplicaSet. It will validate that the cpu and memory resource request fields on the containers in the pod have a value defined. When the policy validation fails, it will include the message defined in the rule.

We'll add an Application to Argo CD that points to the directory in the repo containing the policy.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: policies
spec:
  destination:
    name: kind
    namespace: kyverno
  project: default
  source:
    path: policies
    repoURL: https://github.com/morey-tech/argo-cd-kyverno.git
    targetRevision: main
    directory:
      recurse: true

With the policies synced into the cluster, we will deploy another Argo CD Application containing two nginx pods, one that will pass policy validation and the other that will fail due to missing resource requests.

The Sync will ultimately fail because the policy is set to block any resources that contain a violation from being created in the cluster. Looking at the Sync Status in Argo CD, the user can see in the message which Kyverno policy is failing and the message from the rule explaining why.

The Sync Status will also break down the result for each resource, making it clear to the user which resource failed and why.

On the policy resource, the events will display the violation as well.

Once the policy violation is fixed for the resource, the Argo CD Application will then be able to sync the resource into the cluster.

Conclusion

Kyverno, as a policy engine, is a powerful tool for providing clear feedback to the users of an internal developer platform on achieving the best practices set out by the Platform team. However, this is only the tip of what Kyverno is capable of. The example in this post demonstrates the admission control functionality, but policies can also mutate, generate, and clean up resources. This creates the option to enforce policies by automatically updating resources as they enter the cluster or building a self-service platform that responds to resources created in the cluster to create new clusters and bootstrap them using Argo CD.

Managing the policies for a cluster is simplified when adhering to GitOps principles and using Argo CD to manage the deployment. The Argo CD UI provides a clear understanding of what resources are blocked by policy validations, making it easy for users to understand how to bring the resources into compliance.

Together, Kyverno and Argo CD open up a whole new world of cluster administration that can scale with your organization's needs. For more best practices in using Argo CD and Kyverno head to "3 Essential Tips for Using Argo CD and Kyverno" on Nirmata's blog.

Get in Touch

Are you using Kyverno and Argo CD together? Check out these Argo CD policies. You can find the Kyverno community the Kubernetes Slack.

I'm interested in learning about your Argo CD use-cases! Get in touch with me (Nicholas Morey) on the CNCF Slack. You can also find me in the #argo-* channels.