Are you running security tests in your CI?

You might be wondering — what does running security tests even mean? What does it do? Security tests just test your code for known vulnerabilities, to make sure hackers will not be able to hack into your system. This might sound complicated — but actually, it is pretty simple. There are many existing tools that you can use for running security tests — and in this blog post, I will introduce one of them.

At Soluto, we have a few open source projects, and the biggest one is Tweek. Tweek is a feature management solution, it allows you to do things like A/B test or use feature flags (e.g. open a feature to only part of your users to see how it behaves). It is a critical component of our infrastructure, and therefore, make sure it is secure is really important. Also, as it is an open source project, so anyone can contribute code to it — which could introduce new vulnerabilities. Read along to find out how I easily added security tests to Tweek — or if want to skip the explanation, you can read the PR.

What is Zap and how it can help us

To build our security tests, I used a tool named Zaproxy or just Zap. Zap is an open source and a free hacking tool developed and maintained by OWASP. The coolest part about Zap is that it has an API — which means that I can run all of its hacking functionality in the CI.

The basic feature of Zap is that it functions as a web proxy — e.g. it inspects the traffic from the client to the server. Zap can inspect the request and response, and look for various security issues, like missing security headers. So, I can use existing tests (either UI automation or integration tests), proxy them through Zap and then query Zap for the alerts — let’s see how!

Proxy Tweek’s API smoke tests

Tweek is a complex system, composed of many micro-services and UI. In this blog post, I’ll focus on Tweek’s API, but the same methodology can also be applied to Tweek’s UI. Tweek’s API (and the integration tests) are developed in .NET core (C#). All the integration tests are using HttpClient to send requests, and we can easily configure a proxy:

var baseUrl = Environment.GetEnvironmentVariable("TWEEK_API_URL") ?? "http://localhost:4003";
var proxyUrl = Environment.GetEnvironmentVariable("PROXY_URL");
var handler = new HttpClientHandler();
if (proxyUrl != null)
{
    handler.Proxy = new WebProxy(proxyUrl, false);
}
output.WriteLine($"TWEEK_API_URL {baseUrl}");
var client = new HttpClient(handler);
client.BaseAddress = new Uri(baseUrl);
return new TweekApi(client);

Now, all we need to do is run the tests and look for interesting findings in Zap. Keep in mind that .NET will not proxy requests to localhost — so this will not work when running Tweek locally.

Bringing it all together

Tweek is a containerized app, developed and deployed using Docker. As Tweek contains multiple micro-services, I am using Docker-compose to run Tweek locally. Docker-compose allows you to define all the containers you need in one single yaml file, and then it takes care of the rest. This is the relevant part of the docker-compose yaml:

api:
 image: soluto/tweek-api
 logging:
   driver: "none"
zap:
  image: owasp/zap2docker-bare
  command: zap.sh -daemon -host 0.0.0.0 -port 8090
smoke-tests:
  build:
    context: ../../
    dockerfile: TweekApiSmokeTestDockerfile
  depends_on: 
    - api
    - zap
  working_dir: /repo/tweek
  environment: 
    - TWEEK_API_URL=http://api/
    - PROXY_URL=http://zap:8090
  command: /bin/bash -c "wget --tries 20 --timeout=15 --read-timeout=20 --waitretry=30 --retry-connrefused http://api/status && /src/services/api/Tweek.ApiService.SmokeTests/test.sh"

I omitted all the irrelevant services and tests. You can take a look at the full file here (Tweek has multiple compose files, I referenced the relevant one). In this snippet, you can see we have 3 services:

• The API (using Tweek’s official image ) running Tweek’s API code
• Smoke tests (using test container, this is the dockerfile) which run our test code
• Zap (using Zap’s official bare image — small image, ideal for CI).

You might also notice that the smoke tests are configured to run against our Tweek API, and proxy the requests using Zap. The test.sh script just runs the tests, and queries Zap for interesting alerts — we will deep dive into it soon.
So now all we need to do is run:

docker-compose up --build --exit-code-from smoke-tests

And watch the magic happen. Docker-compose will set the exit code according to the tests and Zap results — so if we have a bug or a security issue, the exit code will be set to 1 — which means we can fail the build! Hurray!

Disabling rules

So now let’s take a look at the results. When I first run the tests through Zap, I noticed that Zap has many alerts. Most of them are because of missing X-Content-Type and Cache-Control headers. I thought that those missing headers were not real vulnerabilities for Tweek API, so I wanted to ignore them. Luckily, Zap's API allows to disabling rules (you can find all the rules here). Let’s take a look at the test.sh script file to see how:

#!/bin/bash
# Abort script on error
set -e
./wait-for-it.sh api:80 -t 4000
./wait-for-it.sh zap:8090 -t 4000
# Disable X-Content-Type scanner - not relevant for API
curl --fail http://zap:8090/JSON/pscan/action/disableScanners/?zapapiformat=JSON\&formMethod=GET\&ids=10021
#Disable Cache control scanner - not relevant for this API
curl --fail http://zap:8090/JSON/pscan/action/disableScanners/?zapapiformat=JSON\&formMethod=GET\&ids=10049
dotnet test

The script uses wait-for-it.sh to wait until Zap and Tweek’s API complete loading. Then, I disabled those two rules that I discussed before using Zap’s API. Now, I can simply call dotnet test, which will run our smoke tests and proxy them through Zap. Now let's continue to the next part - querying Zap for the alerts and failing the build.

Filtering results with Glue

So disabling those rules was easy, but that is not enough. There are cases when you want to ignore or postpone specific issues, but not turn off the entire rule. This was the case with Zap’s last two findings. Zap reported requests that indicate usage of Base64 and SHA-1 — those are the cases that need manual review. When I reviewed Tweek’s case, I found it was clearly a false positive — so I wanted to ignore those findings, but not turn off the rule.

Luckily for us, we have another open source project from OWASP that can help with that — Glue. Glue aims to ease “gluing” security tools into a CI/CD pipeline. It already has support for more than 15 security tools, including Zap. Glue also supports various filtering — like filtering false positives. It also supports various types of reporting — like reporting to Jira. It is open source, so you can easily add more filters or reporters. For example, you can report each new finding to Slack, open a GitHub issue, or report in TeamCity format.

One of the built-in filters in Glue is the file filter — it will write all of the findings to a JSON file, and you can set, for each finding, whether it should be ignored or postponed. You can checkout Tweek’s glue.json file. You’ll notice that I marked those two issues as ignore.

So now let’s take a look at the end of the test.sh file:

ruby /usr/bin/glue/bin/glue
  -t zap
  --zap-host http://zap --zap-port 8090 --zap-passive-mode
  -f text
  --exit-on-warn 0
  http://api
  --finding-file-path /usr/src/wrk/glue.json

Here, I tell Glue that I want to run only Zap, without any other tool. I provide it the host and port that Zap is listening on. Then, I can specify that I want a text report (other reporters currently supported are JSON and Jira) and that I want to set the exit code to 1 on any finding (0 is the lowest level, 3 is the highest). The next parameter is our target — Tweek’s API URL (“http://api"). And the last parameter is our glue.json file, with the ignored findings.
Now if we run it using docker-compose, the exit code will be 0 — no findings! That means we can run it on each new commit — knowing that Zap is watching our back.

How can you use it?

As you saw, this was pretty simple — you can also check out the PR I created in Tweek’s repo to add those security tests. In addition, there is a sample repo demoing this solution — using OWASP’s vulnerable Web App called “Juice Shop”. This repo demonstrates how you can add security tests to Juice Shop — and since it is a vulnerable app, there are some findings. You can clone this repo and play with it to get a better understanding of how this solution works.
I hope you now understand how you can easily add security tests to your CI — but in case something was unclear, feel free to reach out to me. Twitter is the best option, but you can also leave a comment here — I would love to help you integrate Zap into your CI.

Orignaly posted on Soluto Engineering Blog.