Yesterday was released Gradle 8.7. Our repositories are roaring with bot-generated PRs helping us with the processes of updating with the latest and the greatest versions of our dependencies:

Despite those tools being great for automating the updates for small/sample repositories or non-critical dependencies, applying a performance regression test when updating critical build components in highly modularized projects with hundreds of developers is strongly recommended. Something that hurts the performance in this kind of repos will affect the team's development cycle and early detection is crucial. Ultimately, it is too late to detect the issue once the change is merged.

As a simple example, today we will apply a performance test within an experiment on the new Gradle version in the project nowinandroid. The goal is to verify that the update has no impact on our codebase.
In the experiment, we will cover the worst-case scenario where all tasks are executed and we don't have build-cache available.

The project under test, nowinandroid, is still working with Gradle 8.5, so we will add a new variant in the experiment representing Gradle 8.6. So the variants of the experiment are:

• Gradle 8.5
• Gradle 8.6
• Gradle 8.7

The execution environment is:

• Linux 6.5.0-1016-azure (amd64) (GHA runner)
• 4 CPU cores
• 4 Gradle workers
• JDK 17
• 6 GB Gradle Process
• 6 GB Kotlin process

We executed 100 iterations for each variant, each iteration executed the task assembleRelease in a clean GHA runner.

Results
The first obvious check is the overall build time(seconds):

We obtained similar results noticing a 1.78% improvement on the median using 8.7.

Because of the nature of our experiment, fresh agent and no build/remote cache, next we analyzed exclusively the execution time to reduce the noise caused by components like the network(seconds):

Everything looks good, with a decrease of 1.93% in the median of 8.7.

Next, we will focus on the more expensive tasks by plugin. First, we will start with the AGP and the task :app:minifyDemoReleaseWithR8(ms):

We don't observe any significant impact on the task duration and the overall change related to the main median is -0.6%.

Another task that dominates the build times is the DexMergintTask. In nowinandroid the longest task is :app-nia-catalog:mergeExtDexRelease, let's see the results(ms):

All good. We don't observe any impact in the update.

Let's move to the Kotlin Gradle Plugin. In the main branch, the task with the longest duration is :core:model:compileKotlin:

What's going on? Why may the new Gradle version bring benefits in terms of the Kotlin compiler tasks? Sadly, Gradle 8.7 doesn't hide magical optimizations for our Kotlin tasks. The reason is the embedded Kotlin compiler has been updated from 1.9.10 to Kotlin 1.9.22 in Gradle and is now aligned with the version used in the nowiandroid repository.
That means the Gradle build doesn't need to download additional dependencies required for 1.9.22 because they are embedded. And that's why we are seeing an improvement in the task, which is the first Kotlin task executed outside the build logic in the project.
We could have a clear picture if we analyze the build dependencies and network metrics for a build on each variant:

Finally, we analyzed the usage of the processes involved in the build starting with the Gradle process(Gb):

And for the Kotlin process(Gb):

We noticed an increase in process usage caused by the fact that we have the Kotlin versions aligned and we require only one process. In previous versions, two Kotlin processes were created during the build. We could verify this behavior if we analyze the Kotlin processes available at the end of the build for Gradle 8.5/8.6:

Against the processes in Gradle 8.7:

Final words
When updating components like the AGP, KGP, Gradle, or additional critical build components, a performance test regression is recommended to verify the correct behavior of the new version introduced. Even in the case, we explained that it doesn't bring significant duration improvements, it will give us an understanding of not-so-visible changes like the embedded Kotlin compiler update.
The article was just an example covering a few metrics. Depending on the update type and the processes involved in the development cycle, you may consider different tests.

Happy Building!