ABOUT GENNOVACAP

At Gennovacap, we are Open Source Software advocates and supporters. In fact, we are always experimenting with new open source Kubernetes CI CD tools to improve our devops consulting practice. When deploying large scale software systems, we utilize a variety of Kubernetes CI CD tools to help client scale to the moon.

In this article we’ll outline the benefits, features, and architecture of each tool we’ve utilized. We’ll cover the different types of Kubernetes CI CD tools which includes GitOps, pipelines, delivery systems, and deployment extension tools to manage deployments to your Kubernetes clusters.

If you would like to see a full list of Paid Continuous Integration and Continuous Delivery tools that work with Kubernetes check out G2.

Estimated reading time: 15 minutes

TABLE OF CONTENTS

• About Gennovacap
• Kubernetes CI CD Prerequisites
• Devtron: Software Delivery Workflow for Kubernetes
• Skaffold: Easy and Repeatable Kubernetes Development
• Flagger: Progressive Delivery Kubernetes Operator
• Spinnaker: Open-source, Multi-cloud Continuous Delivery Platform
• Werf: Gitops Delivery Tool
• Flux: Open And Extensible Continuous Delivery For Kubernetes
• Argo CD: Declarative Kubernetes Deployment Tool
• Tekton: Cloud Native Pipeline Resource
• Jenkins X: Automated CI CD for Kubernetes
• Kubevela: Kubernetes CI CD Tool for Modern Applications
• Ksonnet: Kubernetes CI CD Tool for multiple Clusters

KUBERNETES CI CD PREREQUISITES

If you have not set up Kubernetes yet, here are some instructions to get you started.

• Setup a Kubernetes Production Environment
• Setup a Kubernetes Container Runtime

You can also setup Kubernetes clusters with KubeAdmin, Kops, or Kubespray. Once you’ve got Kubernetes deployed and your applications are setup in your version control system, you’re all set to wire up CI CD from your applications to Kubernetes. We will not walk through the Kubernetes setup in this article, but you can join our mailing list to receive technical guides on these topics.

DEVTRON: SOFTWARE DELIVERY WORKFLOW FOR KUBERNETES

Devtron

DESCRIPTION:

Devtron is an open-source release orchestration & Kubernetes CI CD tool. It leverages popular Open-Source tools to provide a No-Code SaaS like experience for Kubernetes.

FEATURES:

• No code self-serve DevOps platform
• Multi-cloud/Multi-cluster deployment
• Built-in SecOps tools and integration
• UI-enabled Application debugging dashboard
• Enterprise grade security and compliances
• Automated Gitops based deployment using argocd

LICENSE:

https://github.com/devtron-labs/devtron/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

SKAFFOLD: EASY AND REPEATABLE KUBERNETES DEVELOPMENT

Skaffold

DESCRIPTION:

Skaffold is a command line tool that facilitates continuous delivery for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters. Skaffold handles the workflow for building, pushing and deploying your application. It also provides building blocks and describes customizations for Kubernetes.

FEATURES:

Blazing fast local development

• optimized source-to-deploy: Skaffold detects changes in your source code and handles the pipeline to build, push, and deploy your application automatically with policy based image tagging
• continuous feedback: Skaffold automatically aggregates logs from deployed resources and forwards container ports to your local machine

Project portability

• share with other developers – Skaffold is the easiest way to share your project with the world: git clone and skaffold run
• context aware – use Skaffold profiles, user level config, environment variables and flags to describe differences in environments
• Kubernetes CI CD building blocks – use skaffold run end-to-end, or use individual Skaffold phases to build up your CI CD pipeline. skaffold render outputs hydrated Kubernetes manifests that can be used in GitOps workflows.

Pluggable, declarative configuration for your project

• skaffold init – Skaffold discovers your files and generates its own config file
• multi-component apps – Skaffold supports applications consisting of multiple components
• bring your own tools – Skaffold has a pluggable architecture to integrate with any build or deploy tool

Lightweight

• client-side only – Skaffold has no cluster-side component, so there is no overhead or maintenance burden
• minimal pipeline – Skaffold provides an opinionated, minimal pipeline to keep things simple

LICENSE:

https://github.com/GoogleContainerTools/skaffold/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

FLAGGER: PROGRESSIVE DELIVERY KUBERNETES OPERATOR

Flagger

DESCRIPTION:

Flagger is a Kubernetes deployment tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests. Progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments)

BENEFITS:

• Reduce the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics like HTTP/gRPC request success rate and latency.
• Shift and route traffic between app versions using a service mesh like Istio, Linkerd, OSM or AWS App Mesh. Or if a service mesh does not meet your needs, use an Ingress controller like Contour, Gloo, NGINX, Skipper or Traefik.
• Besides the builtin metrics checks, you can extend your application analysis with custom metrics and webooks for running acceptance tests, load tests, or any other custom validation.

LICENSE:

https://github.com/fluxcd/flagger/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

SPINNAKER: OPEN-SOURCE, MULTI-CLOUD CONTINUOUS DELIVERY PLATFORM ()

Spinnaker

DESCRIPTION:

Spinnaker provides application management and deployment to help you release software changes with high velocity and confidence. Spinnaker is an open-source, multi-cloud continuous delivery platform that combines a powerful and flexible pipeline management system with integrations to the major cloud providers. If you are looking to standardize your release processes and improve quality with this Kubernetes deployment tool, Spinnaker is for you.

BENEFITS:

• Create deployment pipelines that run integration and system tests, spin up and down server groups, and monitor your rollouts. Trigger pipelines via git events, Jenkins, Travis CI, Docker, CRON, or other Spinnaker pipelines.
• Create and deploy immutable images for faster rollouts, easier rollbacks, and the elimination of hard to debug configuration drift issues. Leverage an immutable infrastructure in the cloud with built-in deployment strategies such as red/black and canary deployments.
• Deploy across multiple cloud providers including AWS EC2, Kubernetes, Google Compute Engine, Google Kubernetes Engine, Google App Engine, Microsoft Azure, Openstack, Cloud Foundry, and Oracle Cloud Infrastructure, with DC/OS coming soon.

FEATURES:

• Role-based Access Control
• Chaos Monkey Integration
• CI Integrations
• CLI for Setup and Admin
• Deployment Strategies
• Restricted Execution Windows
• Manual Judgments
• Monitoring Integrations
• Notifications
• VM Bakery

LICENSE:

https://github.com/spinnaker/spinnaker/blob/master/LICENSE.txt

KUBERNETES CI CD TOOL ARCHITECTURE:

Spinnaker CICD Tools

WERF: GITOPS DELIVERY TOOL

Werf

DESCRIPTION:

Werf is an Open Source CLI tool written in Go, designed to simplify and speed up the delivery of applications. To use it, you need to describe the configuration of your application (in other words, how to build and deploy it to Kubernetes) and store it in a Git repo — the latter acts as a single source of truth. In short, that’s what we call GitOps today.

Werf is not a complete Kubernetes CI/CD tool, but a solution for creating pipelines that can be embedded into any existing CI/CD system. It literally “connects the dots” to bring these practices into your application. We consider it a new generation of high-level CI/CD tools.

BENEFITS:

• Werf builds Docker images using Dockerfiles or an alternative fast built-in builder based on the custom syntax. It also deletes unused images from the Docker registry.
• Werf deploys your application to Kubernetes using a chart in the Helm-compatible format with handy customizations and improved rollout tracking mechanism, error detection, and log output.

FEATURES:

• Full application lifecycle management: build and publish images, deploy an application to Kubernetes, and remove unused images based on policies.
• The description of all rules for building and deploying an application (that may have any number of components) is stored in a single Git repository along with the source code (Single Source Of Truth).
• Build images using Dockerfiles.
• Alternatively, werf provides a custom builder tool with support for custom syntax, Ansible, and incremental rebuilds based on Git history.
• Werf supports Helm compatible charts and complex fault-tolerant deployment processes with logging, tracking, early error detection, and annotations to customize the tracking logic of specific resources.
• Werf is a CLI tool written in Go. It can be embedded into any existing CI/CD system to implement CI/CD for your application.
• Cross-platform development: Linux-based containers can be run on Linux, macOS, and Windows.

LICENSE:

https://github.com/werf/werf/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

FLUX: OPEN AND EXTENSIBLE CONTINUOUS DELIVERY FOR KUBERNETES

Flux

DESCRIPTION:

Flux is a tool for keeping Kubernetes clusters in sync with sources of configuration (like Git repositories), and automating updates to configuration when there is new code to deploy.

Flux version 2 (“v2”) is built from the ground up to use Kubernetes’ API extension system, and to integrate with Prometheus and other core components of the Kubernetes ecosystem. In version 2, Flux supports multi-tenancy and support for syncing an arbitrary number of Git repositories, among other long-requested features.

Flux v2 is constructed with the GitOps Toolkit, a set of composable APIs and specialized tools for building Continuous Delivery on top of Kubernetes.

BENEFITS:

• Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built in.
• Enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
• Works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
• Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
• Uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
• Provides health assessments, alerting to external systems, and external events handling. Just “git push”, and get notified on Slack and other chat systems.
• Flux is a CNCF Incubating project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
• We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

FEATURES:

• Describe the entire desired state of your system in Git. This includes apps, configuration, dashboards, monitoring, and everything else.
• Use YAML to enforce conformance to the declared system. You don’t need to run kubectl because all changes are synced automatically.
• Everything is controlled through pull requests. Your Git history provides a sequence of transactions, allowing you to recover state from any snapshot.
• Declaratively configurable using Custom Resources, state reports in the object’s status and via Kubernetes Events, and integrations with Kubernetes RBAC
• Support for e.g. Kustomize, Helm; GitHub, GitLab, Harbor and custom webhooks; notifications to most team communication platforms; and many more.
• Easily create a continuous delivery solution with only the components you need, or use the GitOps Toolkit to extend Flux.

LICENSE:

https://github.com/fluxcd/flux2/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

ARGO CD: DECLARATIVE KUBERNETES DEPLOYMENT TOOL

Argo

DESCRIPTION:

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Benefits/Overview:

FEATURES:

• Automated deployment of applications to specified target environments
• Support for multiple config management/templating tools (Kustomize, Helm, Ksonnet, Jsonnet, plain-YAML)
• Ability to manage and deploy to multiple clusters
• SSO Integration (OIDC, OAuth2, LDAP, SAML 2.0, GitHub, GitLab, Microsoft, LinkedIn)
• Multi-tenancy and RBAC policies for authorization
• Rollback/Roll-anywhere to any application configuration committed in Git repository
• Health status analysis of application resources
• Automated configuration drift detection and visualization
• Automated or manual syncing of applications to its desired state
• Web UI which provides real-time view of application activity
• CLI for automation and CI integration
• Webhook integration (GitHub, BitBucket, GitLab)
• Access tokens for automation
• PreSync, Sync, PostSync hooks to support complex application rollouts (e.g.blue/green & canary upgrades)
• Audit trails for application events and API calls
• Prometheus metrics
• Parameter overrides for overriding ksonnet/helm parameters in Git

LICENSE:

https://github.com/argoproj/argo-cd/blob/master/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

TEKTON: CLOUD NATIVE PIPELINE RESOURCE

Tekton

DESCRIPTION:

The Tekton Pipelines project provides k8s-style resources for declaring CI/CD-style pipelines.

FEATURES:

• Run on Kubernetes
• Have Kubernetes clusters as a first class type
• Use containers as their building blocks
• One Pipeline can be used to deploy to any k8s cluster
• The Tasks which make up a Pipeline can easily be run in isolation
• Resources such as git repos can easily be swapped between runs
• The concept of typed resources means that for a resource such as an Image, implementations can easily be swapped out (e.g. building with kaniko v.s. buildkit)

LICENSE:

https://github.com/tektoncd/pipeline/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

JENKINS X: AUTOMATED CI CD FOR KUBERNETES

Jenkins

DESCRIPTION:

Jenkins X is an automated Kuberentes deployment tool with Preview Environments on Pull Requests using Tekton, Knative, Lighthouse, Skaffold and Helm. It is a Kubernetes deployment tool for modern cloud applications. Jenkins X automates and accelerates Continuous Integration and Continuous Delivery for developers on the cloud, so they can focus on building awesome software.

Embracing popular open source projects Jenkins X automates the setup and management to provide an integrated Cloud Native solution teams can use to develop better software faster and more reliably that traditional non cloud solutions.

BENEFITS/OVERVIEW:

FEATURES:

• Jenkins X aims to use the cloud well, Kubernetes to host the core services, storage buckets for long term storage, container registries and hosted serivces like secrets managers. All of this needs to be created and managed. Jenkins X defers to Terraform to setup and manage the Cloud infrastructure needed by Jenkins X.
• Using GitOps means familiar processes can be followed when making any change to the cluster, using reviews, automation, traceability and rollbacks to give better control over consuming changes.
• Jenkins X GitOps works with External Secrets to provide an integrated experience so your secrets source of truth is a secrets manager and the values are replicated into the cluster when needed.
• By default Jenkins X ships with Tekton for a clean declarative cloud native way to describe pipelines. Combined with Lighthouse Jenkins X makes it easy to inherit versioned shared pipeline steps via Git and a simple syntax providing flexibility and easy maintenance.
• With the ever growing number of microservices needing automation, Jenkins X provides the ability to interact with pipelines via comments on pull requests.
• Along with ChatOps mentioned above Jenkins X aims to help developers have a consistent way of working with their microservices, using a CLI or GUI developers can leverage proven approaches recomended by the Accelerate book.

LICENSE:

https://github.com/jenkins-x/jx/blob/main/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

KUBEVELA: KUBERNETES CI CD TOOL FOR MODERN APPLICATIONS

Kubevela

DESCRIPTION:

KubeVela is a modern application delivery platform that makes deploying and operating applications across today’s hybrid, multi-cloud environments easier, faster and more reliable. KubeVela is infrastructure agnostic, programmable, yet most importantly, application-centric.

FEATURES:

• KubeVela introduces Open Application Model (OAM) as the consistent yet higher level API to capture and render a full deployment of microservices on top of hybrid environments. Placement strategy, traffic shifting and rolling update are declared at application level. No infrastructure level concern, simply deploy.
• KubeVela models application delivery as DAG (Directed Acyclic Graph) and expresses it with CUE – a modern data configuration language. This allows you to design application deployment steps per needs and orchestrate them in a programmable approach. No restrictions, natively extensible.
• KubeVela works as an application delivery control plane that is fully decoupled from runtime infrastructure. It can deploy any workload types including containers, cloud services, databases, or even VM instances to any cloud or Kubernetes cluster, following the workflow designed by you.

LICENSE:

https://github.com/oam-dev/kubevela/blob/master/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE:

KSONNET: KUBERNETES CI CD TOOL FOR MULTIPLE CLUSTERS

KSonnet

DESCRIPTION:

Built on the JSON templating language Jsonnet, ksonnet provides an organizational structure and specialized features for managing configurations across different clusters and environments.

BENEFITS:

• Save time by minimizing duplicate code and concisely describe configurations with a built-in Jsonnet library for Kubernetes.
• Customize prototypes to meet your needs, and scale your application with a flexible framework.
• Seamlessly deploy across environments and share version controlled configurations across teams as the single source of truth.

FEATURES:

• Easily apply a manifest across multiple cluster environments
• Track configurations with version control in order to reliably manage cluster states
• Reuse common patterns from existing configurations or through an external registry
• Use Jsonnet with the Kubernetes-aware Jsonnet library, ksonnet-lib, to write simplified manifests
• See what’s different from what running on a cluster from your configurations
• Leverage existing charts with all the features behind ksonnet

LICENSE:

https://github.com/ksonnet/ksonnet/blob/master/LICENSE

KUBERNETES CI CD TOOL ARCHITECTURE: