Let's talk about sustainable cloud compuzting! 2022 was the year of increased awareness around the amount of CO2 that global cloud infrastructure releases into the atmosphere. Thanks to such initiatives as the CNCF's Environmental Sustainability TAG and the Green Software Foundation, there is a more coordinated effort to promote building cloud-native products and services that are doing as much as they can to be more sustainable and environment-friendly.
GitOps offers organizations several advantages, such as better coordination, transparency, stability, and system reliability. A key characteristic of GitOps is that the managed systems are continuously reconciled, constantly checking whether there are any changes from the "single source of truth".
GitOps tools enable checking this in intervals and performing reconciliation only when necessary. This enables introducing pre-defined "windows" of reconciliation and, outside of them, completely disabling it during times when there are no significant changes to the source of truth. This inherently lowers the energy the systems use by limiting the time spent processing the cluster state and using less compute power.
But what if we can go beyond this to change the state of the cluster to spin down resources when they aren't being used? Where do we start implementing a sustainable approach to cloud resources on top of GitOps?
kube-green is a project that focuses on spinning down
Deployments (reducing the number of replicas to zero) during hours when they are not being used. To quote the docs:
"How many of your dev/preview pods stay on during weekends? Or at night? It's a waste of resources! And money! But fear not, kube-green is here to the rescue."
The people behind kube-green have put in the effort to calculate how much CO2 is produced by a K8s pod. The project has recently joined the CNCF landscape and is listed in the Scheduling & Orchestration subcategory alongside Kubernetes.
If you are already an Argo CD user and want to implement kube-green, you'll have to ensure they work together.
The kube-green project works by deploying a controller and implementing a new CRD called
SleepInfo resource describes what time and days to “sleep” and “wake up”
Deployments. Then the controller takes that configuration and patches the targeted
Deployments to set the
0 when sleeping. Then returns the
replicas value to its original state when waking up.
There is a fundamental conflict between kube-green and Argo CD; both are designed to change the state of the cluster but with different sources of truth. The challenge is that when kube-green patches the
Deployment, Argo CD will see this as a change in the live state of the cluster that deviates from the desired state in Git.
Fortunately, as a mature GitOps tool, Argo CD has some ways around this, but they have their trade-offs.
I won't cover installing kube-green in this post, but the official documentation has a fantastic guide for testing it out with
kind. For an example of installing it with Argo CD and all of the manifests mentioned throughout this, check out my argocd-example-apps repo.
When the kube-green controller “sleeps” a
Deployment, it patches the resources to set the
spec.replicas to zero. If the replicas are explicitly set in the manifests generated by the
Application managing the
Deployment, Argo CD will see the change and mark the
Application as out of sync.
With the automated sync policy and self-heal enabled, the
Application will immediately revert the changes made by kube-green.
To prevent this behavior, you can use the
ignoreDifferences field on the Argo CD
Application that manages the
Deployment modified by kube-green. This configures the
Application to ignore the
spec.replicas field after the initial creation of the resource. For this example, I set the
Application to ignore changes to the
spec.replicas field of any
Deployment in the
The downside of this solution is that if there is a change to the replicas value of the
Deployment in Git, the
Application won't sync the change to the cluster. If you need to regularly update this value in Git, then the next solution might be a better fit.
Instead of ignoring changes to the
replicas field on
Deployments, an alternative solution is to configure Sync Windows that encompass the time used in the
SleepInfo. This will prevent the automated sync policy from running when kube-green has altered the
In this example, I've configured the
default AppProject with a sync window that fully contains the hours in the
SleepInfo timeframe. The sleep and wake-up times in the
SleepInfo are intentionally offset by one minute to give the Sync Windows a chance to come into effect before kube-green makes changes to the
Now when the kube-green controller patches the
Application will be out of sync, but the automated sync policy won't run.
Then at the wake-up time, just before the end of the sync window, the kube-green controller will restore the original replicas count. An added benefit of Sync Windows is that you can manually sync the
Application to spin the
Deployments back up before the wake-up time if needed.
The downside is that the
Application will show as out of sync. You will need to use a manual selective sync sync to implement any changes unrelated to the
Deployments during the Sync Windows (sleep periods) without restoring the replica count on the
More and more projects and initiatives are leading the way in environmentally sustainable cloud computing. If you want to get involved with the CNCF Environmental Sustainability TAG, check out this KubeCon 2022 talk on the topic.
Also, if you have any ideas on using GitOps to create more sustainable infrastructures, consider posting in the #tag-environmental-sustainability channel on the CNCF Slack or attending the GitOps & Environmental Sustainability SubGroup (GitOps-EnvSus) meeting.
GitOps is rapidly becoming the standard for managing cloud-native ecosystems with Kubernetes. Traditional IaC tools fell short with the rise of Kubernetes…...October 19, 2023
Kargo is a next-generation continuous delivery and application lifecycle orchestration platform for Kubernetes. It builds upon GitOps principles and integrates...October 10, 2023
GitOps principles exist to address the genuine problems of visibility and collaboration when working with a complex system like Kubernetes. They stress the…...