Sustainable GitOps with Argo CD and kube-green

Let's talk about sustainable cloud compuzting! 2022 was the year of increased awareness around the amount of CO2 that global cloud infrastructure releases into the atmosphere. Thanks to such initiatives as the CNCF's Environmental Sustainability TAG and the Green Software Foundation, there is a more coordinated effort to promote building cloud-native products and services that are doing as much as they can to be more sustainable and environment-friendly.

GitOps and Sustainability

GitOps offers organizations several advantages, such as better coordination, transparency, stability, and system reliability. A key characteristic of GitOps is that the managed systems are continuously reconciled, constantly checking whether there are any changes from the "single source of truth".

GitOps tools enable checking this in intervals and performing reconciliation only when necessary. This enables introducing pre-defined "windows" of reconciliation and, outside of them, completely disabling it during times when there are no significant changes to the source of truth. This inherently lowers the energy the systems use by limiting the time spent processing the cluster state and using less compute power.

But what if we can go beyond this to change the state of the cluster to spin down resources when they aren't being used? Where do we start implementing a sustainable approach to cloud resources on top of GitOps?

👋 Hello kube-green!

kube-green is a project that focuses on spinning down Deployments (reducing the number of replicas to zero) during hours when they are not being used. To quote the docs:

"How many of your dev/preview pods stay on during weekends? Or at night? It's a waste of resources! And money! But fear not, kube-green is here to the rescue."

The people behind kube-green have put in the effort to calculate how much CO2 is produced by a K8s pod. The project has recently joined the CNCF landscape and is listed in the Scheduling & Orchestration subcategory alongside Kubernetes.

If you are already an Argo CD user and want to implement kube-green, you'll have to ensure they work together.

Using kube-green with Argo CD

The kube-green project works by deploying a controller and implementing a new CRD called SleepInfo. The SleepInfo resource describes what time and days to “sleep” and “wake up” Deployments. Then the controller takes that configuration and patches the targeted Deployments to set the replicas to 0 when sleeping. Then returns the replicas value to its original state when waking up.

There is a fundamental conflict between kube-green and Argo CD; both are designed to change the state of the cluster but with different sources of truth. The challenge is that when kube-green patches the Deployment, Argo CD will see this as a change in the live state of the cluster that deviates from the desired state in Git.

Fortunately, as a mature GitOps tool, Argo CD has some ways around this, but they have their trade-offs.

• Solution 1: Ignore the replicas field of Deployments in Applications.
• Solution 2: Use Sync Windows to prevent syncs during sleep periods.

I won't cover installing kube-green in this post, but the official documentation has a fantastic guide for testing it out with kind. For an example of installing it with Argo CD and all of the manifests mentioned throughout this, check out my argocd-example-apps repo.

Solution 1: ignoreDifferences

When the kube-green controller “sleeps” a Deployment, it patches the resources to set the spec.replicas to zero. If the replicas are explicitly set in the manifests generated by the Application managing the Deployment, Argo CD will see the change and mark the Application as out of sync.

With the automated sync policy and self-heal enabled, the Application will immediately revert the changes made by kube-green.

To prevent this behavior, you can use the ignoreDifferences field on the Argo CD Application that manages the Deployment modified by kube-green. This configures the Application to ignore the spec.replicas field after the initial creation of the resource. For this example, I set the Application to ignore changes to the spec.replicas field of any Deployment in the sleepme namespace.

The downside of this solution is that if there is a change to the replicas value of the Deployment in Git, the Application won't sync the change to the cluster. If you need to regularly update this value in Git, then the next solution might be a better fit.

Solution 2: Sync Windows

Instead of ignoring changes to the replicas field on Deployments, an alternative solution is to configure Sync Windows that encompass the time used in the SleepInfo. This will prevent the automated sync policy from running when kube-green has altered the Deployments.

In this example, I've configured the default AppProject with a sync window that fully contains the hours in the SleepInfo timeframe. The sleep and wake-up times in the SleepInfo are intentionally offset by one minute to give the Sync Windows a chance to come into effect before kube-green makes changes to the Deployments.

Now when the kube-green controller patches the Deployments, the Application will be out of sync, but the automated sync policy won't run.

Then at the wake-up time, just before the end of the sync window, the kube-green controller will restore the original replicas count. An added benefit of Sync Windows is that you can manually sync the Application to spin the Deployments back up before the wake-up time if needed.

The downside is that the Application will show as out of sync. You will need to use a manual selective sync sync to implement any changes unrelated to the Deployments during the Sync Windows (sleep periods) without restoring the replica count on the Deployments

Conclusion

More and more projects and initiatives are leading the way in environmentally sustainable cloud computing. If you want to get involved with the CNCF Environmental Sustainability TAG, check out this KubeCon 2022 talk on the topic.

Also, if you have any ideas on using GitOps to create more sustainable infrastructures, consider posting in the #tag-environmental-sustainability channel on the CNCF Slack or attending the GitOps & Environmental Sustainability SubGroup (GitOps-EnvSus) meeting.