Codecov bash uploader security issue

How minimizing our attack surface with CircleCI Contexts helped us pass the Codecov bash uploader security issue unharmed

What is Codecov

For those who don’t know Codecov, it is an online platform for hosted code coverage reports and statistics founded in 2014. According to their website, over 29,000 organizations and more than 1 million developers use their tools. The list includes Google, Atlassian, The Washington Post, RBC, and Procter & Gamble Co.

Its main tool, the Bash Uploader, is used to send code coverage reports to the platform. Those reports help developers measure how well their source code is covered by tests. This is the main reason why it is widely used among developers and organizations.

What about the breach?

On April 15th, 2021, Codecov reported in a press release that someone had gained unauthorized access to its script and modified it without its permission.

“Our investigation has determined that beginning Jan. 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users’ continuous integration environments. This information was then sent to a third-party server outside of Codecov’s infrastructure” the company said.

According to Codecov, the hacker gained access to:

Any credential, token, or key that customers passed through their continuous integration runner that would be accessible when the Bash Uploader script was executed;
Any service, datastore, and application code that could be accessed with these credentials, tokens, or keys;
The git remote information (URL of the origin repository) of repositories using the Bash Uploader to upload coverage to Codecov in their continuous integration.

For those potential threats, Codecov is strongly encouraging developers to immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI process.

Why are we safe from that at CloudSkiff?

We use CircleCI to continuously build, test and deploy our open-source drift detection CLI, driftctl. Like most projects, we need to manipulate sensitive environment variables which could mean that this event could impact us.

But, CircleCI provides a simple and secure way to store and share those sensitive environment variables across projects and jobs (or pipelines, or steps in other CI), it is called a context. As an Organization in CircleCI, you can, in your settings, create and manage contexts.

You can use the context key in the workflows section of a project config.yml file to give any jobs access to the environment variables associated with the context.

version: 2.1

# Jobs declaration for build, test and deploy not displayed

workflows:
  build-test-deploy:
    jobs:
      - build
      - test
      - deploy:
          context: restricted-context
          requires:
            - build
            - test

In the example above, only the deploy job has access to the environment variables created in the restricted-context context. On the other hand, the build and test jobs don’t have any access to the sensitive variables in their execution.

That’s basically what we do as of today in our CircleCI config.yml file. Every time we need to call our test job to execute the Codecov Bash Uploader script (e.g. every push, pull request, and release workflows), since we don’t attach any context to it, the hacker has no access to our secret variables.

We try to keep our jobs as small as possible to reduce the attack surface by separating concerns, and CircleCI context is well suited to assist in this good practice.

We could have avoided that threat

One way we could have avoided that is the same way Codecov started investigating the attack. The company reported on the morning of April 1, 2021, that one customer was using the shasum that is available on its Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash.

Indeed, as stated on its documentation website, “it can be validated for correctness by calculating the shasum on download and comparing to the result we store in the GitHub repository”.

An example could be:

$ curl -s -o codecov https://codecov.io/bash \
&& VERSION=$(grep 'VERSION=\".*\"' codecov | cut -d'"' -f2) \
&& shasum -a 512 -c --ignore-missing <(curl -s [https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM)](https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM))

This simple script could be run on each test job before running the Bash Uploader. If we were doing it that way we could have seen sooner the hack, and we could have reported it sooner to Codecov.

Our final test job looks like this:

test:
docker:
  - image: golang:1.16
steps:
  - checkout
  - run: make install-tools
  - run: make test
  - run:
      name: Check Codecov
      command: |
        curl -s -o codecov https://codecov.io/bash \
          && VERSION=$(grep 'VERSION=\".*\"' codecov | cut -d'"' -f2) \
          && shasum -a 512 -c --ignore-missing <(curl -s https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM)
  - run:
      name: Codecov upload
      command: |
        chmod +x codecov
        ./codecov

Driftctl is a free and open-source CLI that tracks, analyzes, prioritizes, and warns of infrastructure drift

More info about us

Reference:

Blog

Codecov bash uploader security issue

driftctl