May 28, 2024
Alexander Matyushentsev
Security-Hardened Image for Argo CD
Argo CD, widely adopted by tens of thousands of developers for managing production workloads on Kubernetes,
inherently requires access to highly sensitive systems and data. Consequently, security is a top priority
for the project. Argo CD maintainers are working hard to collaborate with the community and promptly address
vulnerabilities in the project source code. Argo CD has a very solid process for handling security issues
as well as a dedicated security team that is responsible for triaging and addressing security issues. The project
conducts regular security audits to identify and address potential security issues proactively. Finally, maintainers
are using security scanners to identify vulnerabilities in the source code and the third-party dependencies.
Third Party Dependencies Vulnerabilities
Third-party dependencies are a common source of security vulnerabilities, and Argo CD is no exception. The project depends on a number of third-party components, and that number is quite large. As of today the project has nearly three thousand dependencies that include Golang and Javascript libraries, third-party CLI tools as well as packages bundled with the base container image. All these dependencies are being scanned routinely for vulnerabilities and the maintainers are working hard to keep them up to date. As a result, the most recent Argo CD v2.11.1 has 27 known vulnerabilities inside the third-party dependencies.
So why do vulnerabilities still exist? Well, some just don't have remediation available yet. For example
CVE-2024-2236 in the libgcrypt20 package is still open and unfortunately,
no fix is available for the base image. A vulnerable package does not necessarily make your Argo CD insecure. A lot of factors are
at play, and the attacker cannot just exploit the vulnerability without compromising the system in some other way. So the risk is
low and it does not justify upgrading the base image. Another reason is - the number of vulnerabilities across all dependencies is
just too high to address them all. So the practical approach is to methodically keep the dependencies up to date and address the
most critical ones first. The maintainers are doing a great job at that, and the number of vulnerabilities stays steady low from
release to release.
The problem with this approach is that security is not an area where we can stop at "good enough". We need to be proactive and thrive for the best possible security. At Akuity we've decided to take a different approach and address the problem at the root: minimize the number of dependencies and reduce the attack surface.
Security-Hardened Argo CD Distribution
A lot of detected vulnerabilities are part of the base image that Argo CD is built on. The base image is an Ubuntu version 24.04. Ubuntu was chosen over Debian because it has a more frequent release cycle, and the packages are more up-to-date. The shorter release cycle is still not enough to keep up with fixes for all pre-installed packages. Since a lot of packages are not needed for Argo CD we've decided to switch the based image to Distroless - the image that contains absolutely necessary runtime dependencies. The remaining packages are installed on top of the Distroless along with the Argo CD binaries. This dramatically reduces the number of dependencies and the attack surface.
After initial testing, we've discovered that Distroless image supports all Argo CD features and actually minimizes the number of vulnerabilities. For example, Distroless based Argo CD distribution has only one known vulnerability in the installed kustomize version, compared to 27 known vulnerabilities in the off-the-shelf distribution of Argo CD.
Vulnerabilities
| Package | ID | Severity | Installed Version | Fixed Version |
|---|---|---|---|---|
kustomize | CVE-2024-24788 | HIGH | 5.4.1-r3 | 5.4.1-r4 |
Continuous Maintanence And Security Monitoring
Our team is committed to keeping the security-hardened Argo CD distribution up to date and secure. We've automated the process of producing the Distrolless based images for each Argo CD version. So whenever a new version of Argo CD is released, we are able to produce the security-hardened image in a matter of minutes.
The images are signed using sigstore that provides the proof of the image origin and the integrity of the image.
Finally, we are using security scanners to monitor the image for vulnerabilities and provide visibility into the security posture of the Argo CD distribution. The security scanning report is available inside the Akuity Documentation section on security-hardened Argo CD and provides information about the latest patch releases of the three most recent stable versions.
What is Next?
Fostering software security is a never-ending process, and we are constantly looking for ways to improve the security of Argo CD. As the next step, we are considering is further hardened Distroless-based images including the following enhancements:
- Include the Helm and Kustomize versions with the security patches into the security-hardened images.
- Security-hardened images for the Akuity version of Argo CD — an enhanced Argo CD version with additional features such as server-side pagination.
- Support of hardened images inside the Akuity Platform.
Please give us your feedback and influence which of these should be implemented first. We are looking forward to hearing from you about what are your needs regarding the Security-Hardened Argo CD images, straight from the creators of the Argo Project.
