Application Dependencies with Argo CD

With Argo CD and GitOps gaining wide adoption, many organizations are starting to deploy more and more applications using Argo CD and GitOps in their workflows. As adoption grows many organizations are on-ramping more complex deployment patterns as applications are being converted from the "lift and shift" model to being refactored with cloud-native architecture in mind. Seeing the velocity it brings, folks aren’t only adopting GitOps, but also start to use Argo CD as a mechanism to migrate into Kubernetes.

Although many organizations are reaching a mature level of adoption, there are still growing pains as companies move their workloads onto containers, Kubernetes, and Argo CD. Even after moving away from the "lift and shift'' model, there is still a need to do more fine-grained deployment patterns. Not everything is as immutable as we’d like it to be and we at times still need a method to couple individual application deployments on Kubernetes.

Keeping that in mind, it’s no surprise that one of the most prevalent questions that come up when using Argo CD is "How does Argo CD handle Application dependencies?"

In this blog we will go over everything you need to know in order to set up Application dependency management using Argo CD.

Enter the Application

Those that have been using Argo CD for a while, know that Argo CD has an abstraction called the "Application". An Argo CD Application can be seen as a collection of Kubernetes resources (Deployments, Services, Ingress, etc) that act as a single entity. It’s the atomic unit of work for Argo CD and it’s used to not only make sure related resources are deployed together, but also to handle the lifecycle management of these related resources.

By default, Argo CD applies manifests "as-is". This works, except in the case where order matters (for example, you want a CustomResourceDefinition applied before the corresponding CustomResource). To that end, Argo CD has the ability to control how these manifests get applied with Sync waves and phases.

Sync phases focus on things like pre and post sync hooks where Sync waves focus on which order to apply manifests. A simple example is applying a Namespace before a Pod.

apiVersion: v1
kind: Namespace
metadata:
  name: web
  annotations:
    argocd.argoproj.io/sync-wave: "1"
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  name: nginx
  namespace: web
spec:
  containers:
  - image: nginx
    name: nginx
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always

While Sync waves are a great way to order how manifests get applied, this only works for the resources contained in an Argo CD Application - but not the Argo CD Application itself, or between Applications. For more information about Sync phases and Sync waves, see the official Argo CD documentation.

So what if I have multiple Argo CD Applications? For example, I have an Application called database that I want deployed and healthy before deploying an Argo CD Application called api. Unfortunately there’s no "native" way (that is, no way built into the Argo CD Application spec itself) to do this. However, it’s still possible to setup Argo CD Application dependencies with readily available tools and methods inside the Argo CD ecosystem.

The methods we are going to cover are:

• Eventual Consistency
• App-of-Apps Pattern
• ApplicationSets Progressive Sync

Before we get into these, there are some considerations you need to keep in mind that we will go over first.

Important Considerations

There are some important considerations to take into account when implementing any of the solutions outlined in this blog. Because of that, we’ve decided to consolidate them in one session. One reason being that you’ll probably need to implement these regardless which method you go with. The second reason being these are generally best practices when working with Argo CD.

So before diving into how to handle Argo CD Application dependencies, we’ll go over some prerequisites and best practices. These include Readiness/Liveness probes, Argo CD Application Health Checks, and Resource Health Checks.

Set up Probes

It’s generally good practice to set up readiness and liveness probes for those of your Kubernetes manifests that support them. For those who aren’t familiar with the concept, liveness probes check to see if your resource (like a container in your Deployment) is up and running (aka "alive"); readiness probes check to see if your resources are ready to accept connections. For more information about readiness and liveness probes, take a look at the official Kubernetes documentation.

Setting up readiness and liveness probes is not only a best practice, it’s paramount to an Argo CD Application. Argo CD Application health is based on the collective health of the bundle of manifests being deployed. Without proper readiness and liveness probes, Argo CD might mark resources as "Healthy" and "Synced" when in fact, it might still be trying to deploy.

Let’s take the scenario of deploying a MySQL database. If we deploy the MySQL StatefulSet without any probes, Argo CD will mark the MySQL StatefulSet as "healthy" even though it might be going through its setup process. Furthermore, it will also be marked as "healthy" when the StatefulSet isn’t even ready to start receiving requests! To that end, you can see how adding probes can help when deploying something with Argo CD. Here is an example of adding probes for MySQL:

spec:
  template:
    spec:
      containers:
        - image: mysql:5.6.51
          name: mysql
          livenessProbe:
            tcpSocket:
              port: 3306
            initialDelaySeconds: 12
            periodSeconds: 10
          readinessProbe:
            exec:
              command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
            initialDelaySeconds: 12
            periodSeconds: 10

In this example, Kubernetes considers the MySQL StatefulSet as "alive" when the port is listening for connections and it will consider it "ready" when you’re able to run a query.

Argo CD Health Checks

Argo CD doesn’t only rely on the generic Kubernetes health status for the objects it’s managing, but it also provides built-in health checks for a multitude of Kubernetes types, which are then surfaced to the overall Application health status as a whole. Health checks are written in Lua, and you can see the current built-in checks in the Argo CD GitHub repo.

There are times where there’s a need to add or customize these health checks. For example, if you’re working with a Kubernetes Operator (perhaps because you have either written one for your organization or because you’re using a relatively new one), you might need to add these custom health checks in the resource.customizations field in the argocd-cm ConfigMap. The format looks like the following:

data:
  resource.customizations: |
    <group/kind>:
      health.lua: |

For example, here is what the health check for the cert-manager.io/Certificate object would look like in the argocd-cm ConfigMap.

data:
  resource.customizations: |
    cert-manager.io/Certificate:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.conditions ~= nil then
            for i, condition in ipairs(obj.status.conditions) do
              if condition.type == "Ready" and condition.status == "False" then
                hs.status = "Degraded"
                hs.message = condition.message
                return hs
              end
              if condition.type == "Ready" and condition.status == "True" then
                hs.status = "Healthy"
                hs.message = condition.message
                return hs
              end
            end
          end
        end

        hs.status = "Progressing"
        hs.message = "Waiting for certificate"
        return hs

To read more about Argo CD Health Checks please refer to the official documentation.

Application Health

Another important thing to note is that the health check for the Argo CD Application CRD has been removed in Argo CD 1.8 (see issue #3781 for more information). This is an important thing to keep in mind, especially in the case of doing Argo CD Application dependencies. Since some of the patterns we’re going to go through rely on the Argo CD Application health check’s presence, we‘ll need to add it to the argocd-cm ConfigMap. This is easily done. Here’s an example:

data:
  resource.customizations: |
    argoproj.io/Application:
      health.lua: |
        hs = {}
        hs.status = "Progressing"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            if obj.status.health.message ~= nil then
              hs.message = obj.status.health.message
            end
          end
        end
        return hs

With all these considerations (not only are they general best practices, but they’re also prerequisites for the next sections) in place, we can start exploring different patterns on how to get Argo CD Application dependencies.

Eventual Consistency

One of the patterns that can be used is, ironically, to not use any dependency management at all and, instead, rely on the fact that things will eventually be consistent with retries. This can easily be setup using the Argo CD Application manifest itself and also by leveraging Argo CD Sync Option annotation. Here’s an example Application manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-go
spec:
  destination:
    name: in-cluster
    namespace: demo
  source:
    path: deploy/overlays/default
    repoURL: 'https://github.com/christianh814/simple-go'
    targetRevision: main
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - Validate=false
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m0s
        factor: 2

Notice under .spec.syncPolicy.syncOptions that the Validate=false option is there. This disables resource validation (equivalent to kubectl apply --validate=false). There are also retries set in this section to tell Argo CD to retry when an error occurs. You can (and probably should) also add the following annotation to resources that depend on others existing first (like a CR of a CRD)

metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

Note that a "dry run" will be performed if the dependent resource is there.

These two settings together, will make Argo CD "keep retrying until successful or until the retries are exhausted" (whichever comes first). In this way, Argo CD handles dependencies by not handling them; and instead keeps trying to apply them.

There are some drawbacks to this pattern. The biggest one is that disabling validation can lead to eventual consistency never happening because you are trying to deploy an invalid manifest. However, more importantly, some things just cannot happen before others. An example of this is installing Istio and making sure Istio is up and running so that it can inject sidecars in your application before your application starts.

The App of Apps Pattern

Originally convinced as a method of bootstrapping Argo CD, the App of Apps pattern is basically an Argo CD Application that consists of other Argo CD Applications (Since an Argo CD Application is nothing but a Kubernetes CRD). Bootstrapping and having a need to deploy Argo CD Applications using Argo CD itself was where this pattern originated from.

Extending beyond just bootstrapping, users found other advantages of using this pattern thanks to also having access to other features that Argo CD gives you like Sync waves and Sync Phases. When setting up probes and Argo CD Application Health, you will now have everything you need to set up Application dependencies!

Let’s take a look at an example of deploying a 3 tiered application. We will have one Argo CD Application that deploys a frontend app, a backend app, and also a database. We want to have these managed by a "parent" Argo CD Application and we want to deploy these in the following order.

1. Database
2. Backend
3. Frontend

In order to do this we’ll have to use Sync Waves with our App of Apps. We first annotate the database. Keep in mind that lower numbers get higher priority.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  name: database
  namespace: argocd

Since we want the backend to come up afterwards, we’ll annotate that Application with a higher number.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  name: backend
  namespace: argocd

Finally, we annotate the frontend Application with a higher number than the database and backend so that it comes up last.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "3"
  name: frontend
  namespace: argocd

You can see an example of this in this repo.

The "parent" Application is just another Argo CD Application, and there’s nothing "special" about it except the fact that it’s deploying other Argo CD Applications. Here’s the example we are using:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: parent
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  source:
    path: argocd/applications
    repoURL: 'https://github.com/christianh814/app-of-apps-example'
    targetRevision: main
  destination:
    namespace: argocd
    name: in-cluster
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m0s
        factor: 2
    syncOptions:
      - CreateNamespace=true

Once this "parent" Argo CD Application is applied, Argo CD will apply the "child" Argo CD Applications in order it was annotated with.

Walking through this, you’ll see:

• First, with the database (since it’s annotated with a "1")
• Once that’s synced and healthy, Argo CD will apply the "backend" (since it’s annotated with a "2")
• Once that’s synced and healthy, Argo CD will finally apply the "frontend" (since it’s annotated with a "3")
• In the end, all 3 Applications are synced!

As you can see, this is a powerful way of setting up Application dependencies and it’s, currently, the recommended way of doing it. It’s worth reiterating that this works because all readiness/liveness probes were set up and Argo CD was configured with the proper LUA health checks.

Progressive Syncs

With all the power that Argo CD gives you with Argo CD Applications and the App-of-Apps pattern, there was still a need to templatize the creation of Argo CD Applications. Yes we can manage Argo CD Application deployments in a controlled manner, but we still needed to create those Application manifests. A lot of end users used Helm for this, until the creation of Argo CD ApplicationSets.

Argo CD ApplicationSets can be seen as an Application "factory". The sole purpose of the Argo CD ApplicationSet controller is to create Argo CD Applications. This gives us the ability to not only create multiple Applications at the same time using a single manifest, it also allows us to deploy many applications to many destination clusters. How the ApplicationSet controller generates Argo CD Applications depends on which "generator" is used. You can read more about generators in the official Argo CD documentation.

The one drawback of ApplicationSets is that it just generates Applications. There was no built-in mechanism to order or have dependencies. That was until ApplicationSets ProgressiveSyncs was introduced.

The ProgressiveSync feature aims to deploy the Applications in an ApplicationSet in the specified order, with keeping Application health into account (meaning it won’t create Applications unless the previous one is synced and healthy). While a great way to use ApplicationSets ProgressiveSyncs, there are a few things to keep in mind:

1. Generated Applications will have autosync disabled.
2. This is an Alpha feature and will be subject to change. This also means that it needs to be explicitly enabled.
3. If an Application has been in a "Pending" state for more that the allotted progressing timeout (default 300 seconds) the ApplicationSet controller will mark it as "Healthy"

To read more about ProgressiveSyncs, please see the official Argo CD documentation.

Even with ProgressiveSyncs enabled, you still need to set up your readiness/liveness probes and Argo CD Application health. With all these things in mind, let's go over the same example as before, except with ProgressiveSync. The same repo I mentioned earlier has an example that can be looked at.

Here’s an example of the ProgressiveSync we are using:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: golist
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - srv: database
        path: apps/golist-db/
      - srv: backend
        path: apps/golist-api/
      - srv: frontend
        path: apps/golist-frontend/
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: golist-component
              operator: In
              values:
                - database
        - matchExpressions:
            - key: golist-component
              operator: In
              values:
                - backend
        - matchExpressions:
            - key: golist-component
              operator: In
              values:
                - frontend
  template:
    metadata:
      name: '{{srv}}'
      labels:
        golist-component: '{{srv}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/christianh814/app-of-apps-example'
        targetRevision: main
        path: '{{path}}'
      destination:
        name: in-cluster
        namespace: golist
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
        syncOptions:
          - CreateNamespace=true

Once applied, it will create all 3 of the Argo CD Applications at once; but they will remain "missing / out of sync". Take a look at the progression:

You’ll notice:

• It will first start syncing the first Argo CD Application, which is the database Application in our case.
• Once the database is deployed, the backend Application will start syncing.
• When the backend Application is deployed, the frontend Application will begin to sync.
• In the end, it should look the same as the "App of Apps" did.

The result is the same; except that with ProgressiveSyncs; there is only one manifest to apply.

It’s important to note that it’s not App of Apps vs ProgressiveSyncs. There are some situations that come up where you use both or a combination of both (Like "App of AppSets" or "AppSet of App of Apps"). It varies from organization to organization.

Looking into the Future

While it’s possible to set up Argo CD Application dependencies with current Argo CD tools, it doesn’t mean that a "dependsOn" feature isn’t valid. There is work going on in Argo CD to bring this feature "natively" in the Argo CD Application controller. Lots of discussion is going on around this and you can follow by looking at issue 7437.

Also, it’s worth reiterating that ProgressiveSyncs are still an Alpha feature so a lot of teams are hesitant to implement them. Although seemingly stable, I would take caution with putting this feature on a production system and I would mainly have it running on test systems.

We at Akuity are at the forefront of GitOps and are always looking for ways to improve the end user experience with not only Argo CD, but with GitOps in general. That’s why we’ve created a new Open Source project called Kargo; which aims to not only aid in GitOps promotions, but also help with dependencies as well.

Conclusion

In this blog we went over all the things you need to know in order to handle Application dependencies with Argo CD. We looked at different patterns like eventual consistency, App of Apps, and progressive syncs. We also went into the prerequisites and best practices around these patterns.

If you are interested in more GitOps best practices, go and download the GitOps Best Practices Whitepaper! Also, come join the Akuity community Discord server where we also share tips and tricks and information around the Akuity Platform, Argo CD, and Kargo. Join now by visiting https://akuity.community. New to the Akuity platform? Come see how you can supercharge your Argo CD installation by signing up for our free trial!