Testing Gradle plugins can be challenging. It relies on a special-purpose framework, TestKit, that is only relevant for this singular use-case—so unless you've already heard of it, you wouldn't have heard of it. But now you definitely have, so welcome to the club! The other main challenge is grokking the nature of Gradle itself, which is that it's all about manipulating files. So much so, in fact, that an alternate title for this post may have been: Files all the way down.

I'm going to say it one more time because I think it's a critical point on the road to Gradle mastery. In these days of reactive programming, with streams of values chained together with functions operating on strongly-modeled data types—your build system operates on capital-F Files. Tasks write their outputs to the filesystem, other tasks read those files in, and terminal tasks generate distributable artifacts—jars, aars, apks, etc.—on the filesystem.

What has this got to do with testing Gradle plugins? Read on and see!

What to expect if you continue reading

This is an intermediate-level post that will demonstrate how to set up comprehensive testing infrastructure for a Gradle plugin. The techniques you'll see below are in use right now in CI pipelines used to validate the plugins that build industrial-scale software, for companies that earn billions in revenue on that back of that software. Perhaps more importantly for you as an individual, they form part of a core skillset that will make you seem like a very valuable wizard. Recruiters beware!

All the code below is available on Github.

The plugin under test

While the focus of this post will be on how to test your Gradle plugins, we need something to test. Let's get that out of the way:

class MeaningOfLifePlugin : Plugin<Project> {

  override fun apply(project: Project) {
    project.tasks.register(
      "meaningOfLife",
      MeaningOfLifeTask::class.java
    ) { t ->
      t.output.set(
        // in practice, this typically resolves to
        // "build/meaning-of-life.txt"
        project
          .layout
          .buildDirectory
          .file("meaning-of-life.txt")
      )
    }
  }
}

@UntrackedTask(because = "No meaningful inputs")
abstract class MeaningOfLifeTask : DefaultTask() {

  init {
    group = "Gradle All the Way Down"
    description = "Computes the meaning of life"
  }

  @get:OutputFile
  abstract val output: RegularFileProperty

  @TaskAction fun action() {
    // Clean prior output. nb: Gradle ensures this file
    // exists, since it's been annotated with @OutputFile
    val output = output.get().asFile.also { it.delete() }

    // Do a computation
    val meaningOfLife = DeepThought().meaningOfLife()

    // Write output to disk
    output.writeText(meaningOfLife.toString())
  }
}

internal class DeepThought {
  fun meaningOfLife(): Any {
    // ...long-running computation...
    return 42
  }
}

I don't intend to focus very much on this code, except to reiterate once again that our task's output is some text written to disk. Conceptually, most Gradle tasks should fall into the "pure function" bucket. That is, given a set of inputs, the task will perform some "heavy" computation, and then emit one or more outputs to disk. If the inputs are stable, then the outputs should be as well—if they're not, that's either a bug in Gradle or (more likely) a bug in your code.¹ In our case, we happen to have a task that has no meaningful input, and so we annotate it with @UntrackedTask, an annotation available since Gradle 7.3. This serves as both documentation and also a flag to Gradle that it should skip input/output hashing, which has a positive performance impact.

Testing strategy

We're going to follow a two-pronged testing strategy for this code. For demonstration purposes, we'll have some simple unit tests, both for our plugin and our DeepThought computational powerhouse. More importantly, we will setup a functional testing suite using TestKit that will invoke a real Gradle build and then validate the (file) output on disk. Along the way, we'll try to have some fun with testing frameworks and the filesystem.

The build script

You didn't think you'd be able to get away with reading a post about Gradle and not see a single build script, did you? The script below goes beyond the minimum necessary for setting up a basic TestKit test, but I've done so explicitly because I want the example to be closer to production code than a simpler example could manage. The code below would pass PR review on my team.

I have added somewhat verbose comments inline, but nevertheless I will go into some depth on a few items in just a moment.

plugins {
  // Write Gradle plugins...
  id 'java-gradle-plugin'
  // ...in Kotlin...
  id 'org.jetbrains.kotlin.jvm'
  // ...and test them with Groovy
  id 'groovy'
}

dependencies {
  implementation platform('org.jetbrains.kotlin:kotlin-bom')
  implementation 'org.jetbrains.kotlin:kotlin-stdlib-jdk8'
}

// Configure our test suites (an @Incubating feature)
testing {
  suites {
    // Configure the default test suite
    test {
      // JUnit5 (JUnit Jupiter) is the default
      useJUnitJupiter()
      dependencies {
        // Equivalent to `testImplementation ...` in the
        // top-level dependencies block
        implementation 'com.google.truth:truth:1.1.3'
      }
    }

    // Create a new test suite
    functionalTest(JvmTestSuite) {
      useSpock()
      dependencies {
        // functionalTest test suite depends on the
        // production code in tests
        implementation project
        implementation 'com.google.truth:truth:1.1.3'
        implementation 'com.autonomousapps:testkit-truth:1.1'
      }
    }
  }
}

// Define our plugin
gradlePlugin {
  plugins {
    greeting {
      id = 'mutual.aid.meaning-of-life'
      implementationClass = 'mutual.aid.MeaningOfLifePlugin'
    }
  }

  // TestKit needs to know which source set to use. nb: this must 
  // come below `testing`, because that is what creates our
  // functionalTest DSL objects.
  testSourceSets(sourceSets.functionalTest)
}

// Groovy code can depend on Kotlin code
def compileFunctionalTestKotlin = tasks.named('compileFunctionalTestKotlin')
tasks.named('compileFunctionalTestGroovy', AbstractCompile) {
  dependsOn compileFunctionalTestKotlin
  classpath += files(compileFunctionalTestKotlin.get().outputs.files)
}

The two most unusual aspects of this (I think!), are the testing block and the final block which sets up a compilation dependency between Groovy and Kotlin. Let's talk about the first, first.

The testing DSL is added by the JVM Test Suite Plugin, which is added automatically by the java plugin (which is added automatically by the java-gradle-plugin plugin). This plugin and its DSL are @Incubating, but I think it's worth giving a look thanks to two primary features: (1) the ease with which it lets build authors set up additional test suites, which has always been painful; and (2) the fact that it enables the Test Aggregation Report Plugin, which solves the vexing problem of aggregating test reports across multiple subprojects in a multi-project build. While we don't have a multi-project build in our sample scenario, I expect most of us work on multi-project builds.²

We've done two things with testing:

1. Configured the default test suite (named "test", i.e., the "unit tests") to use the JUnit5 (aka JUnit Jupiter) test framework, and to add Truth as a testImplementation dependency.
2. Added a new test suite, named "functionalTest", and configured it to use the Spock test framework, along with three dependencies: Truth, TestKit-Truth, and the project itself. This reveals the interesting point that, by default, new test suites don't have the project under test on the classpath, which enables true black box testing if you're into that kind of thing.

The final block, which sets up the dependency from compileFunctionalTestKotlin to compileFunctionalTestGroovy, is not strictly necessary but I've added it here because it's a useful concept. First, to answer the question that some of you have probably been screaming at me: yes, we will be writing a test in Groovy, using the Spock framework.

I will die on this hill. That said, I respect the fact that many of my readers' first, last, and only language is Android-flavored Kotlin, so we're going to write the minimum amount of Groovy possible, and it'll be great, I promise.

Unit tests are de rigueur

For brevity, I will not discuss how the unit tests work in the sample repo. Instead, I encourage you to take a look if you're curious. Bottom line: exactly like you'd expect, with maybe the small exception that it uses JUnit5 rather than 4. I've included it primarily to demonstrate the possibility and utility of maintaining multiple test suites that each serve different purposes.

Functional tests with TestKit

Below I include all of the Groovy code in this project.³ In my (perhaps controversial) opinion, Spock is a powerful, expressive, and Really Very Good framework. That said, I think this is also straightforward enough that if you really hated it, you could rewrite it in Kotlin and you'd only lose some of this expressiveness and power. I've included the import statements for clarity, since this test (or spec, as Spock prefers) makes use of two external libraries in addition to Spock itself, and some custom wiring code we'll go over in a moment.

This code lives at src/functionalTest/groovy.

import mutual.aid.fixture.AbstractProject
import mutual.aid.fixture.MeaningOfLifeProject
import mutual.aid.gradle.Builder
import org.gradle.util.GradleVersion
import spock.lang.AutoCleanup
import spock.lang.Specification

import static com.autonomousapps.kit.truth.BuildTaskSubject.buildTasks
import static com.google.common.truth.Truth.assertAbout
import static com.google.common.truth.Truth.assertThat

class MeaningOfLifePluginSpec extends Specification {

  // AbstractProject implements AutoCloseable
  @AutoCleanup
  AbstractProject project

  def "can determine meaning of life (#gradleVersion)"() {
    given: 'A Gradle project'
    project = new MeaningOfLifeProject()

    when: 'We ask for the meaning of life'
    def taskPath = ':meaningOfLife'
    def result = Builder.build(gradleVersion, project, taskPath)

    then: 'Task succeeded'
    assertAbout(buildTasks())
      .that(result.task(taskPath))
      .succeeded()

    and: 'It is 42'
    def actual = project.buildFile('meaning-of-life.txt').text
    assertThat(actual).isEqualTo('42')

    where:
    gradleVersion << [
      GradleVersion.version('7.3.3'), 
      GradleVersion.version('7.4')
    ]
  }
}

There are three patterns embedded in the above code that I really like:

1. BDD (given/when/then) provided by Spock itself.
2. Data-driven testing, also from Spock, for easy parameterization of tests.
3. A clean separation between the project-under-test (encapsulated by the AbstractProject class and its sub-classes) and the spec that drives it.

I use the above patterns extensively in the testing strategy for the Dependency Analysis Gradle Plugin, and it has enabled me to radically rewrite that plugin's implementation while leaving the specs themselves almost totally untouched.

It's worth talking about the assertions at some length. Let's look more closely:

then: 'Task succeeded'
assertAbout(buildTasks()).that(result.task(taskPath)).succeeded()

and: 'It is 42'
def actual = project.buildFile('meaning-of-life.txt').text
assertThat(actual).isEqualTo('42')

The first section simply asserts that the given task (":meaningOfLife") was successful. The assertion itself is driven by Truth and a custom extension to it I've written called testkit-truth that lets us write fluent assertions about the result of a Gradle build.

The second section is more interesting, to my mind, and gets to the heart of what I mean when I say that Gradle's primary abstraction is the filesystem itself. We know that our plugin produces a file output with the path build/meaning-of-life.txt. So, we run a build, and then directly read the file produced by that build, and assert is has the expected value. There are two things that are very powerful about this strategy: (1) these are not unit tests, so they are completely agnostic as to implementation. We can rewrite the code that does our expensive "meaning of life" computation, and these tests will still serve to prevent regressions. This makes the time spent developing the testing infrastructure a good long-term investment: no, or very little, churn. (2) If we have any issues with our test suite, one very useful debugging strategy is to inspect the generated code directly! Those files really exist on disk, and if you navigate to them in your IDE, it will provide syntax highlighting. We'll explore that some more in the next section.

Fixtures, Kotlin-style

As I said, that's the only Groovy in the project. All of the imports point either to external libraries or custom test fixtures written in Kotlin, and which live at src/functionalTest/kotlin.

First, we have a wrapper around GradleRunner, versions of which I include in all of my projects. I won't go into detail as to what each method call accomplishes, and instead encourage you to read the javadoc linked above. Suffice it to say, if you call build() with the appropriate arguments, the net effect will be a real Gradle build is executed inside your test. Gradle calling Gradle, with the results returned as a BuildResult for asserting against.

object Builder {
  @JvmOverloads
  @JvmStatic
  fun build(
    gradleVersion: GradleVersion = GradleVersion.current(),
    project: AbstractProject,
    vararg args: String
  ): BuildResult = runner(
    gradleVersion,
    project.projectDir,
    *args
  ).build()

  private fun runner(
    gradleVersion: GradleVersion,
    projectDir: Path,
    vararg args: String
  ): GradleRunner = GradleRunner.create().apply {
    forwardOutput()
    withPluginClasspath()
    withGradleVersion(gradleVersion.version)
    withProjectDir(projectDir.toFile())
  }
}

This next bit of infrastructure is pure custom test fixture code. This is not the only way to do it, but is about as concise as I could make it while still being "production-ready."

// AutoCloseable means Spock will call its close() method
// if we annotate instances with @AutoCleanup.
abstract class AbstractProject : AutoCloseable {

  val projectDir = Path.of("build/functionalTest/${slug()}")
    .createDirectories()
  private val buildDir = projectDir.resolve("build")

  fun buildFile(filename: String): Path {
    return buildDir.resolve(filename)
  }

  private fun slug(): String {
    val worker = System.getProperty("org.gradle.test.worker")?.let { w ->
      "-$w"
    }.orEmpty()
    val className = javaClass.simpleName
    val uuid = UUID.randomUUID().toString().take(16)
    return "${className}-${uuid}$worker"
  }

  override fun close() {
    projectDir.parent.toFile().deleteRecursively()
  }
}

class MeaningOfLifeProject : AbstractProject() {

  private val gradlePropertiesFile = projectDir.resolve("gradle.properties")
  private val settingsFile = projectDir.resolve("settings.gradle")
  private val buildFile = projectDir.resolve("build.gradle")

  init {
    // Yes, this is independent of our plugin project's 
    // properties file
    gradlePropertiesFile.writeText("""
      org.gradle.jvmargs=-Dfile.encoding=UTF-8
    """.trimIndent())

    // Yes, our project under test can use build scans. 
    // It's a real project!
    settingsFile.writeText("""
      plugins {
        id 'com.gradle.enterprise' version '3.8.1'
      }

      gradleEnterprise {
        buildScan {
          publishAlways()
          termsOfServiceUrl = 'https://gradle.com/terms-of-service'
          termsOfServiceAgree = 'yes'
        }
      }

      rootProject.name = 'meaning-of-life'
    """.trimIndent())

    // Apply our plugin
    buildFile.writeText("""
      plugins {
        id 'mutual.aid.meaning-of-life'
      }
    """.trimIndent())
  }
}

To best understand what this fixture accomplishes, I think it's simplest to inspect the files generated when we run our spec. First, make a slight change and run the spec:

You'll note we've commented out @AutoCleanup, to stop Spock from running our fixture's close() method. There are other ways to accomplish this, but sometimes quick-and-dirty wins the race. Click the run button, and then expand the build directory in the IDE's project view:

You'll note we have two basically identical sets of generated files, only differing by the name of the root directory, which of course is the combination of a UUID + a system property provided by Gradle from the slug() function above. We can now open one of the build.gradle files and see that it's exactly what we defined in MeaningOfLifeProject, complete with syntax highlighting!

I often use this technique when a spec is failing and it's unclear why. It can be much faster to visually inspect the generated files vs debugging.

We can also see the results of running our spec:

and the console output from one of the tests:

> Task :meaningOfLife

BUILD SUCCESSFUL in 3s
1 actionable task: 1 executed

Publishing build scan...
https://gradle.com/s/s7rw4ig672sa6

That's right, that's a build scan from our artificial project-under-test (Gradle all the way down). For such a simple scenario as we're testing, it's not very interesting, but I have found this to be an invaluable resource when testing more complex plugins run against more complex artificial projects.

Wrapping up

In this post, we've learned how to set up a functional testing suite for our Gradle plugins, using Gradle TestKit, the Spock testing framework, and the Truth and TestKit-Truth assertions libraries. Following the pointers in the code, you should be able to use JUnit4 or 5 as you prefer, or other assertion libraries. I may return for a follow-up post with a more complex example demonstrating how to more thoroughly manipulate the environment of the build-under-test, including how to inject AGP versions for testing against multiple releases in the same test invocation. Stay tuned!

Endnotes

¹ Or, and let's be frank, a Gradle footgun. up
² At time of writing, this plugin only works for the JVM, not Android. up
³ Well, not counting the Gradle script. up