Cloudsmith and Buildkite

At Cloudsmith, you will often hear us refer to our mantra of “Automate Everything”. It's a quest that we never deviate from, and we believe that anything that can be automated, should be automated.

With that in mind, we would like to show you how simple it is to integrate a Cloudsmith repository with your Buildkite pipeline, and automate the pushing of your build artifacts into your own private repository for further CI/CD steps or even as a source for your global distribution needs.

What is Buildkite?

Buildkite is a platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure. That means you can use Buildkite to orchestrate and manage your own fleet of build hosts, and these can be anything from containers, to cloud instances, to bare metal servers.

Why would I want to integrate Cloudsmith with Buildkite?
Well, in short, a continuous integration pipeline is going to have an output, and you are going to need and want to put that output somewhere that is secure, controlled and integrates with all the other tooling that will form the next parts of your DevOps workflow - whether that is continuous deployment to production, distribution to an end-user or customer, or even consumption by another internal team as part of their development process.

This is where Cloudsmith fits in, and this is another phrase you might hear us use quite a bit - Cloudsmith offers a central source of truth for your packages and build artifacts. We provide a global platform that gives you the performance, scalability, security and visibility that you need to control and manage your software assets.

Using Buildkite and Cloudsmith – An Example:

So, you’re let’s assume that you’re new to Buildkite, how do you get started?

Step 1- Install the buildkite-agent

The first thing that you need to do is install the buildkite-agent so that you have a machine (remember, this can be a container, a VM, or even a real server) that will act as your build host. Buildkite provides install instructions for all major platforms and operating systems:

Once you have installed the agent, you will see a new build host in the Buildkite UI:

For this example, I have installed the buildkite-agent on a Debian instance.

Step 2(a) – Create a pipeline

The next thing you need to do is create your pipeline. A Buildkite pipeline is a series of steps that your build host(s) will execute in order to build your assets/artifacts. For this example, we will create a pipeline that will compile a simple C source program, package it into a deb package and then push that deb package to a Cloudsmith repository.

To get started, you give your pipeline a name, and then you specify a source repository for the pipeline. In this case, it is a GitHub repository (although Buildkite will also work with many other platforms as a source):

If using a private GitHub repository, you also need to remember to add an SSH key to your GitHub profile so that the host you have installed the buildkite-agent on can access the source repository.

2(b) – Add your pipeline steps.

Pipeline steps are where you specify the commands or scripts that you need to run in order to build your source / project / application. In Buildkite, you can define these steps via the Buildkite UI or as a pipeline.yaml file. To keep things simple, we will define two steps via the Buildkite UI.

PreBuild Step

In this step, we install the tools we need to build our source and package it into a deb package. We also install the Cloudsmith CLI:

BuildAndPushPackage Step

In this step, we use make to compile our source and then we use fpm to build the deb package. Finally, we use the cloudsmith push command to push the package to our Cloudsmith repository:

The equivalent pipeline.yaml file for these steps would look like:

steps:

  - label: "PreBuild"
  commands:
      - sudo apt update
      - sudo apt-get install ruby ruby-dev rubygems build-essential python-pip -y
      - sudo gem install --no-document fpm
      - pip install cloudsmith-cli 

  - label: "BuildAndPushPackage"
    commands:
      - make
      - fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-buildkite-test .
      - cloudsmith push deb demo/buildkite-demo/debian/buster cloudsmith-buildkite-test_1.0.1_amd64.deb

For more complex build pipelines, you’ll likely have a lot more steps and the advantage of using a pipeline.yaml file is that you can version it and check it in right alongside your source.

One thing to note is that pipeline steps in Buildkite are stateless. As a result, if you have a fleet of agents then each step is not guaranteed to run on the same agent. This means that if a subsequent step need / relies on the output from a previous step, the output from one step will need to be stored and then retrieved. Buildkite provides temporary artifact storage that you can use for this purpose (see here for more details), but to keep things simple in this example we performed the build and push in a single step.

Step 2(c) – Add a Github Webhook.

We want this pipeline to start when we push a new commit of our source to our GitHub repository and for that, we can configure a GitHub Webhook. Conveniently, Buildkite provide a webhook URL that we just need to copy and paste into our GitHub webhook configuration, and then select the event type we wish to fire this webhook on (in this case, all pushes):

That’s it, our pipeline is now built and will be ready to run.

Step 3 – Environment hooks

Buildkite supports several types of hooks that can run on a build host during a pipeline run and a final piece of configuration that we need to do is use an environment hook to configure our environment.

As we are using the Cloudsmith CLI to push the package to our Cloudsmith repository, we need to set up our Cloudsmith API Key as an environment variable. We do this because we don’t want to store a sensitive secret like an API-Key in our pipeline source, or within a build step as a plain text environment variable where it could be exposed in logs. There are other alternative methods of managing secrets in Buildkite, see here for more details.

Our environment hook is pretty simple:

set -euo pipefail

if [[ "$BUILDKITE_PIPELINE_SLUG" == "cloudsmith-buildkite-demo" ]]; then
    export CLOUDSMITH_API_KEY="abcdefghijklmnop1234567890"
    export PATH="$HOME/.local/bin:$PATH"

fi

It does two things:

• Sets the CLOUDSMITH_API_KEY environment variable
• Adds the location that the Cloudsmith CLI is installed to our PATH Additionally, it only runs when executed by a pipeline with the name “cloudsmith-buildkite-demo”

OK, at this stage our pipeline is built and ready, and the environment on our build host will be set up when the pipeline executes. It’s time to test it out.

Step 4 – Push a change to our source repository.

Now if we make a change our source, commit it and then push the change to our GitHub repository, our build will start:

And once the BuildAndPushPackage step has completed, we can view the output in the logs:

We can see that our deb package was successfully built and pushed to our Cloudsmith repository.

If we now go to the repository on the Cloudsmith website, we can see the built package:

To sum up

Buildkite is a very powerful and flexible CI management platform, and integrating your Cloudsmith repositories with your Buildkite pipelines is as simple as installing the Cloudsmith CLI and then using the Cloudsmith push command in your build steps. We would really encourage you to give it a go yourself, our trial is fully-featured and we are here to help you get started. No question is too big or too small.

Happy Build(kite)ing!