This week, Code Intelligence released CI Fuzz CLI, an open-source tool for Java that enables automated fuzz testing within JUnit. In this walkthrough, I want to demonstrate how you can use it to fuzz Java code using literally three commands.

The three commands:

# Initialize fuzzing
$ cifuzz init

# Create your first fuzz test
$ cifuzz create my_fuzz_test

# Run fuzz test and find bugs
$ cifuzz run my_fuzz_test

CI Fuzz CLI is an easy-to-use fuzzing tool that enables you to integrate and run fuzz tests directly from your command line. Fuzz testing is a great add-on to classic unit testing, as it adds an element of negative testing, by feeding software with millions of unexpected or invalid inputs.

CI Fuzz CLI runs on Linux, macOS, and Windows and supports integrations with common build systems and IDEs. If you want to test a Java application with CI Fuzz CLI, the only prerequisites are Java JDK (e.g., OpenDesk or Zulu) and a build system of your choice.

You can download CI Fuzz CLI here or by running the installation script in GitHub.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/CodeIntelligenceTesting/cifuzz/main/install.sh)"

In the README, you will find detailed instructions on how to install the tool.

Fuzzing Use Case: How to Find Bugs in a Java Library

This example library has multiple branches that are reached under different conditions. It contains a potential remote code execution, where an attacker could inject a string and overwrite the settings.

ExploreMe.java
package com.example;
public class ExploreMe {
  // Function with multiple paths that can be discovered by a fuzzer.
  public static void exploreMe(int a, int b, String c) {
      if (a >= 20000) {
          if (b >= 2000000) {
              if (b - a < 100000) {
                  // Create reflective call
                  if (c.startsWith("@")) {
                      String className = c.substring(1);
                      try {
                        Class.forName(className);
                      } catch (ClassNotFoundException ignored) {
                      }
                  }
              }
          }
       }
   }
}

Let's take a look at how to write a fuzz test with CI Fuzz CLI that triggers this remote code execution.

How to Set Up a Fuzz Test in 3 Easy Steps

To set up CI Fuzz CLI, just follow the instruction in the documentation.

How to set up CI Fuzz CLI in Maven

CI Fuzz CLI commands will interactively guide you through the needed options and give you instructions on what to do next. You can find a complete list of commands with all options and parameters by calling cifuzz command --help.

1. Initialize the Project for CI Fuzz CLI

The first step is to initialize the project you want to test with CI Fuzz CLI.

This step will vary depending on which build system you are working with, as you will need to modify relevant build files, either pom.xml for Maven projects or build.gradle for Gradle projects. Creating a fuzz test is the same for each build system. Learn more.

To initialize a Java Maven project, first run cifuzz init in the root directory of the project. This will create cifuzz.yaml in the root directory of the project, the primary configuration file for CI Fuzz CLI.

CI Fuzz CLI will also list the dependencies you need to add to your pom.xml for your project:

<dependency>
  <groupId>com.code-intelligence</groupId>
  <artifactId>jazzer-junit</artifactId>
  <version>0.13.0</version>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-engine</artifactId>
  <version>5.9.0</version>
  <scope>test</scope>
</dependency>

2. Create a Fuzz Test

To create a fuzz test, cifuzz create java. This will create a fuzz test stub in the current directory. You can also use the -o flag to specify a path to create the fuzz test.

You can put the fuzz test anywhere, but we recommend you keep it close to the tested code as you would a regular unit test. In the example maven project in the cifuzz repository, the fuzz test is created in src/test/java/com/example/FuzzTestCase.java.

Here is that fuzz test:

FuzzTestCase.java
package com.example;
Import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;
public class FuzzTestCase {
    @FuzzTest
    void myFuzzTest(FuzzedDataProvider data) {
        int a = data.consumeInt();
        int b = data.consumeInt();
        String c = data.consumeRemainingAsString();
        ExploreMe.exploreMe(a, b, c);
    }
}

If you are familiar with unit tests in Java, this should look mostly familiar. A few things to note about this fuzz test:

• The fuzz test is in the same package as the target class

• The @FuzzTest annotation identifies this method as a fuzztest method

• The FuzzedDataProvider is part of the Jazzer API package and makes it easy to split fuzzing input into multiple parts and various data types.

• The name of this fuzz test, as cifuzz recognizes it, is com.example.FuzzTestCase. So when you want to run this fuzz test, then run cifuzz run com.example.FuzzTestCase from the project directory.

3. Run the fuzz test

Start the fuzzing by executing cifuzz run FuzzTestCase. CI Fuzz CLI now tries to build the fuzz test and starts a fuzzing run.

$ cifuzz run FuzzTestCase 
[...]

Use ‘cifuzz finding <finding name>’ for details on a finding. 

💥[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)

Note: The crashing input has been copied to the seed corpus at: 
    src/test/resources/com/examples/MyClassFuzzTestInputs/awesome_gnu

It will now be used as a seed input for all runs of the fuzz test, including remote runs with artifacts created via ‘cifuzz bundle’ and regression tests. For more information, see:
    https://github.com/CodeIntelligenceTesting/cifuzz#regression-testing

Execution time: 3s
Average exec/s: 316880
Findings: 1
New seeds: 5 (total: 5)

Coverage Reporting and Debugging

What you can do now is take a closer look at the findings. cifuzz findings shows you a list of all findings and cifuzz finding [name] will show you detailed information about a particular finding, which is useful to understand and fix a bug.

Here we have the stack trace for the remote code execution:

[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)
Date: 2022-11-10 13:31:07.94532426 +0100 CET

    == Java Exception: com.code_intelligence.jazzer.api.FuzzerSecurityIssueHigh: Remote Code Execution

Unrestricted class loading based on externally controlled data may allow
remote code execution depending on available classes on the classpath.
    at jaz.Zer.<clinit>(Zer.java:54)
      at java.base/java.lang.Class.forName0(Native Method)
      at java.base/java.lang.Class.forName(Class.java:375)
      at com.example.ExploreMe.exploreMe(ExploreMe.java:13)
      at com.example.FuzzTestCase.myFuzzTest(FuzzTestCase.java:13)
    == libFuzzer crashing input ==
    MS: 0 ; base unit: 0000000000000000000000000000000000000000

0x40,0x6a,0x61,0x7a,0x2e,0x5a,0x65,0x72,0x0,0x1,0x0,0x40,0x0,0x0,0x0,0x75,
    @jaz.Zer\000\001\000@\000\000\000u
    artifact_prefix='./'; Test unit written to 
    ./crash-f29a1020fa966baaf8c2326a19a03b73f8e5a3c9
    Base64: QGphei5aZXIAAQBAAAAAdQ==

We also get a report of the CLI, how many lines and functions were found, and how many of the branches were covered.

Coverage Report
               File |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found
  FuzzTestCase.java |         2/2 (100.0%) |     9/9 (100.0%) |       0/0 (100.0%)
src/explore_me.java |         1/1 (100.0%) |   23/23 (100.0%) |       8/8 (100.0%)
                    |                      |                  |     
                    |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found                                      
              total |                  3/3 |            32    |                8/8

If you can, optimize your fuzzer for code coverage. The idea behind this recommendation is to enable the fuzzer to explore your application by taking as many different paths through your code as possible. This way, the fuzzer will be able to learn if a specific data structure or inputs are required to execute the code, and the more information the fuzzer gets about the structure of the code, the more effective your fuzz tests will be.

Closing Thoughts

Getting started with fuzzing can be challenging, as many open-source fuzzers require a lot of setup and knowledge to get your first fuzz test up and running. But I hope this article and CI Fuzz CLI can make it easier for you to get started with fuzzing, as it’s really one of the most effective (and fun) ways to test Java for integrity, reliability issues, and other vulnerabilities.

If you have any questions about this article or CI Fuzz CLI, please feel free to reach out to my colleague @markuszoppelt(Twitter)!

CI Fuzz CLI on Github: https://github.com/CodeIntelligenceTesting/cifuzz

Our test automation whisperer Josh will host a freely accesible live stream to demo some of the CLI's key features and to answer questions. More info at https://www.code-intelligence.com/webinar.