Reproducible Builds for Supply Chain Security

Why supply chain security matters

In today’s interconnected world, software development relies heavily on third-party dependencies and components. However, using these components introduces a significant risk to the security and integrity of software supply chains. Malicious actors can exploit vulnerabilities in these components to gain unauthorized access to sensitive data or to execute unauthorized code on targeted systems. The recent SolarWinds attack is a prime example of how a supply chain attack can have far-reaching consequences.

To address these concerns, there is an increasing focus on supply chain security, which involves ensuring the security and integrity of software components throughout their entire lifecycle, from development to deployment.

One of the popular topics in the context of supply chain security is reproducible builds. “Reproducible” means repeating the build with the same inputs results in a bit-for-bit identical output. Reproducible builds guarantee that multiple parties can independently build and verify software components, ensuring that the resulting binaries are consistent and free from tampering or malicious code.

How to protect against supply chain attacks

Different solutions have been developed to help organizations identify and track vulnerabilities in their software components and dependencies, allowing them to prioritize and manage their security patches and updates.

A software Bill of Materials (SBOM) records all the software components and dependencies used in a particular software product. SBOMs provide critical visibility into the software supply chain, allowing organizations to track and manage software components and vulnerabilities. We have published dedicated blog posts on SBOMs and how we integrate them into our supply chain security.

An additional approach to protecting against supply chain attacks is implementing the Supply Chain Levels for Software Artifacts (SLSA) framework. SLSA is a set of best practices and recommendations resulting from a cross-industry collaboration by organizations including Google and The Linux Foundation, for ensuring the security and integrity of software components throughout their entire lifecycle, from development to deployment. You can learn more about SLSA in the official documentation.

Why reproducible builds matter for confidential computing

In confidential computing, reproducible builds play a crucial role in the semantic verification of remote attestation statements. Using reproducible builds, organizations can verify that their confidential computing environments are running on trusted software components, free from hidden vulnerabilities or malicious code. Further, they can verify that precisely their intended workload is running before sharing data with it.

Reproducible builds and containers

Containers have become popular for building, packaging, and deploying software applications. However, they also introduce new challenges and risks for supply chain security. Containers typically rely on multiple layers of dependencies, including base images, application code, and third-party libraries. These dependencies can be vulnerable to attacks, and malicious actors can exploit these vulnerabilities to gain access or execute unauthorized code.

For these reasons, it’s essential to implement robust supply chain security measures for containers. One approach is to use container image signing, which involves signing the container image with a digital signature to verify its authenticity and integrity. This ensures that only trusted container images are used in production and helps prevent supply chain attacks.

However, container signing is not enough regarding remote attestation in confidential computing. It would still mean trusting the signing party. The semantic attestation requires that the container is also built reproducibly. That way we can actually verify the containers based on the sources.

Bringing it all together with Constellation

Combining all these measures, we can provide gap-free supply chain security for our confidential Kubernetes platform Constellation. By generating SBOMs, implementing SLSA, and ensuring reproducible builds for our images, containers, and binaries, Constellation provides confidential computing-grade protection against the risks of supply chain attacks. In the following, we describe how we implement reproducible builds in practice using Bazel.

Bazel

Bazel is a powerful and flexible build system that has gained popularity recently for its ability to manage large, complex software projects. Developed and open-sourced by Google, Bazel is designed to handle software builds in a distributed, parallel, and incremental way, making it an excellent tool for large-scale software development. Bazel is designed to support reproducible builds by default, making it an ideal choice for us.

One of the key features of Bazel is its built-in caching system. Bazel stores build artifacts in a cache and can reuse previously built artifacts for future builds, enabling the same build to produce identical outputs across different environments. This is particularly important in a distributed development environment, where developers may work on different machines and operating systems.

Bazel also supports sandboxing, where builds are executed in a controlled and isolated environment, ensuring that builds are not affected by external factors such as system libraries or environment variables. This helps to ensure that builds are reproducible across different development environments.

In addition, Bazel provides deterministic builds by default, meaning that the same build inputs will always produce the same outputs. Bazel uses a content-based addressing system to ensure that build artifacts are uniquely identified by their content rather than by their location or time of creation. This ensures that builds are consistent and reproducible, regardless of the built environment or timing.

Bazel’s support for containerization is built on its existing build infrastructure, enabling developers to define container images as a build target like they define other build targets.

Overall, Bazel’s built-in caching, sandboxing, containerization support, and deterministic builds make it an excellent tool for Constellation’s reproducible builds.

Reproduce a Constellation build

We now walk through the steps of locally reproducing a specific Constellation version’s build. You can find detailed descriptions and instructions in the build and test documentation.

Make sure you have the build dependencies installed.

Download the sources at the right version in this case v2.9.0

git clone https://github.com/edgelesssys/constellation.git && cd constellation
git fetch --all --tags
git checkout tags/v2.9.0

Build the release. For the sake of this demo, we build the CLI and the Constellation service containers
```
bazel build //cli:cli_enterprise_linux_amd64
bazel build //bazel/release:container_sums
```

Verify the official CLI release matches our local build

mkdir build && cd build
cp ../bazel-bin/cli/cli_enterprise_linux_amd64.
wget https://github.com/edgelesssys/constellation/releases/download/v2.9.0/constellation-linux-amd64
sha256sum ./constellation-linux-amd64
# ca657e7552ef4c7c2aba1e7b2fbf7e7ab2dbb07fb3e56918e3ece15f6c6c1dc0  ./constellation-linux-amd64
sha256sum ./cli_enterprise_linux_amd64
# ca657e7552ef4c7c2aba1e7b2fbf7e7ab2dbb07fb3e56918e3ece15f6c6c1dc0  ./cli_enterprise_linux_amd64

Verify the containers running inside Constellation match our local build

chmod u+x ./constellation-linux-amd64
./constellation-linux-amd64 config generate <your_cloud>
# Fill in cloud provider specific information, see the docs for more info https://docs.edgeless.systems/constellation/workflows/config
./constellation-linux-amd64 create -c 1 -w 2
./constellation-linux-amd64 init
export KUBECONFIG="$PWD/constellation-admin.conf"
kubectl get pods -n kube-system -l 'k8s-app in (join-service,key-service,verification-service)' -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.containerStatuses[0].imageID}{"\n"}{end}'
# join-service-tjxpc: ghcr.io/edgelesssys/constellation/join-service@sha256:6783eddfe541bad769461935b0d7b08e9891cc6a94a4398969bb5d22c17b18c2
# key-service-48zx7: ghcr.io/edgelesssys/constellation/key-service@sha256:74ad2ee200cca87057d78f8eec64815bf10e1c33c084c568b011851a32a5e7f9
# verification-service-5h5jt: ghcr.io/edgelesssys/constellation/verification-service@sha256:707123efc4e7fac1c34f35007ab687ac917d24f53bcd5e06244bcd4f569f14a9
kubectl get pods -n kube-system -l app.kubernetes.io/name=constellation-operator -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.containerStatuses[1].imageID}{"\n"}{end}'
# constellation-operator-controller-manager-6ddcd74c5b-wbvxd: ghcr.io/edgelesssys/constellation/node-operator@sha256:a34d20bf823fd308c87e40705dc124969760d6d0810f23dc19a313607ab232ff
cat ../bazel-bin/bazel/release/container_sums.sha256
# 6783eddfe541bad769461935b0d7b08e9891cc6a94a4398969bb5d22c17b18c2  ghcr.io/edgelesssys/constellation/join-service:v2.9.0
# 74ad2ee200cca87057d78f8eec64815bf10e1c33c084c568b011851a32a5e7f9  ghcr.io/edgelesssys/constellation/key-service:v2.9.0
# 707123efc4e7fac1c34f35007ab687ac917d24f53bcd5e06244bcd4f569f14a9  ghcr.io/edgelesssys/constellation/verification-service:v2.9.0

Comparing the hashes, you should see your local builds matching the released CLI and deployed services.

Conclusion

In this post, we have seen the critical role that reproducible builds play in ensuring the security and trustworthiness of software supply chains, particularly in the context of confidential computing. By ensuring that software artifacts are consistent and free from tampering, reproducible builds help to protect against a wide range of security threats and vulnerabilities.

With its built-in support for reproducible builds, we have seen how Bazel is an excellent tool for implementing these principles into Constellation’s build pipeline in practice.

Finally, we have learned hands-on how a Constellation release can be reproduced and verified locally based on the source code.

Blog