If you've landed on this blog post, I assume you're already familiar with Kubernetes and know it's a container orchestration platform. As new application releases are created, you've got your CI processes down for building container images. Kubernetes is fantastic in providing a wide range of functionality that enables users to perform complex deployment strategies. The challenge is that there are good and bad ways to use it, depending on your environment.
The platform engineers in your organization are likely intimately familiar with Kubernetes and understand the nuances of the right and the wrong way to deploy something into the cluster based on the specifics of your organization. The challenge comes from disseminating this information to all of the users of the platform.
Most of the operators' understanding likely came from formal training or spending many hours building the platform and learning from their mistakes. It would be unrealistic to require that all of the application developers that intend to interact with the platform go through the same experience. It's expensive for the organization in terms of time, effort, and potential impact on the product and the customers as these lessons are learned.
Inside every organization, there are policies and standards that users of the internal development platform are aware of or are required to follow. The challenge is that many organizations use documentation and broad communication to ensure users follow these standards. This provides a long feedback loop between people deviating from the expected standards and learning the correct method.
Enter Kyverno, a Kubernetes-native policy engine that provides a way to codify the lessons learned by platform administrators. Kyverno deploys directly into a Kubernetes cluster and uses custom resources for policy definitions. Using admission webhooks, Kyverno can audit policy violations or go as far as blocking requests as they attempt to enter the cluster.
A policy engine can significantly reduce the feedback loop between attempting to apply a change in Kubernetes and learning that deployment does not meet the standards set by the platform's administrators. As application developers interact with the cluster, the policies provide feedback immediately on if they are deviating from the best practices set out by the platform team.
For example, a developer becoming familiar with Kubernetes decides to deploy the
latest tag for their application. This may seem harmless, but they are unaware of the dangers of using a variable tag. Until this mistake is caught by a cluster administrator and reported back to the user, a lot of damage can be done in that time.
Instead, when the user goes to deploy the
latest tag into the cluster, they can be immediately prompted with a clear message explaining why this is not allowed and steps to remediate the policy violation. This enables them to continue with what they were trying to accomplish, deploying their application into the cluster.
Organizations can even integrate policies into the CI/CD workflow to catch any violations earlier in the release cycle. If somehow a violation makes it to the cluster, Kyverno can block the admission request to enforce the policy or even mutate resources as they are applied so that they conform with best practices.
As a Kubernetes administrator, you're probably curious about the best way to manage these policies; this is where Argo CD and GitOps come in.
GitOps brings best practices for application development, such as version control, collaboration, compliance, and tooling, and applies them to Kubernetes. The manifests for Kubernetes resources used in a cluster are stored in Git, which provides an immutable version of the desired state.
Argo CD acts as the GitOps agent, which continuously reconciles the difference between the desired state and the live state of the cluster. It takes any changes in Git and applies them to the Kubernetes cluster.
Like any Kubernetes manifest, Kyverno policies can be stored and managed through Git. This enables them to follow transparent approval processes and provide an audit trail of when policies change and why.
Let us look at a typical example of enforcing Kubernetes best practices with Kyverno policies. We'll start by adding a policy to the cluster using Argo CD. For this example, Kyverno and Argo CD are already installed into the cluster.
In the GitOps repository, add the
require-resource-requests policy, which requires that pods have defined resource requests. This is critical to Kubernetes' ability to schedule pods onto nodes with sufficient resources. Without them, in terms of resource availability, Kubernetes will schedule pods blindly onto nodes.
apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-requests annotations: policies.kyverno.io/title: Require Resource Requests policies.kyverno.io/category: Best Practices policies.kyverno.io/severity: medium policies.kyverno.io/subject: Pod policies.kyverno.io/minversion: 1.6.0 policies.kyverno.io/description: >- As application workloads share cluster resources, it is important to specify resources requested and consumed by each Pod. Requiring resource requests per Pod is recommended, especially for memory and CPU. If a Namespace level request is specified, defaults will automatically be applied to each Pod based on the LimitRange configuration. This policy validates that all containers have something specified for memory and CPU requests. spec: validationFailureAction: enforce background: true rules: - name: validate-resource-requests match: any: - resources: kinds: - Pod namespaces: - dev validate: message: 'CPU and memory resource requests are required.' pattern: spec: containers: - resources: requests: memory: '?*' cpu: '?*'
validationFailureAction field determines the behavior of how Kyverno responds to a failed validation check. The
require-resource-requests policy is set to
enforce a rule failure, blocking any request that violates them. When this happens, it will record a
fail (blocked) event on the policy resource responsible.
validate-resource-requests rule will match any
Pod resource in the
dev namespace, even if generated from a higher-level resource, such as a Deployment or ReplicaSet. It will validate that the
memory resource request fields on the containers in the pod have a value defined. When the policy validation fails, it will include the
message defined in the rule.
We'll add an Application to Argo CD that points to the directory in the repo containing the policy.
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: policies spec: destination: name: kind namespace: kyverno project: default source: path: policies repoURL: https://github.com/morey-tech/argo-cd-kyverno.git targetRevision: main directory: recurse: true
With the policies synced into the cluster, we will deploy another Argo CD Application containing two
nginx pods, one that will pass policy validation and the other that will fail due to missing resource requests.
The Sync will ultimately fail because the policy is set to block any resources that contain a violation from being created in the cluster. Looking at the Sync Status in Argo CD, the user can see in the message which Kyverno policy is failing and the message from the rule explaining why.
The Sync Status will also break down the result for each resource, making it clear to the user which resource failed and why.
On the policy resource, the events will display the violation as well.
Once the policy violation is fixed for the resource, the Argo CD Application will then be able to sync the resource into the cluster.
Kyverno, as a policy engine, is a powerful tool for providing clear feedback to the users of an internal developer platform on achieving the best practices set out by the Platform team. However, this is only the tip of what Kyverno is capable of. The example in this post demonstrates the admission control functionality, but policies can also mutate, generate, and clean up resources. This creates the option to enforce policies by automatically updating resources as they enter the cluster or building a self-service platform that responds to resources created in the cluster to create new clusters and bootstrap them using Argo CD.
Managing the policies for a cluster is simplified when adhering to GitOps principles and using Argo CD to manage the deployment. The Argo CD UI provides a clear understanding of what resources are blocked by policy validations, making it easy for users to understand how to bring the resources into compliance.
Together, Kyverno and Argo CD open up a whole new world of cluster administration that can scale with your organization's needs. For more best practices in using Argo CD and Kyverno head to "3 Essential Tips for Using Argo CD and Kyverno" on Nirmata's blog.
We are thrilled to announce Kargo, a multi-stage application lifecycle orchestrator for continuously delivering and promoting changes through environments…...
The Akuity Platform offers cloud-hosted Argo CD that allows managing hundreds of Kubernetes clusters with no hassle of maintaining and scaling the control plane...