Introduction

Code coverage is the most common metric to measure code quality, but it does not guarantee that tests are testing the expected behavior.

"...100% code coverage score only means that all lines were exercised at least once, but it says nothing about tests accuracy or use-cases completeness, and that’s why mutation testing matters". (Baeldung, 2018)

The idea of mutation testing is to modify the covered code in a simple way, checking whether the existing test set for this code will detect and reject the modifications.

Good tests should fail when your service rules are changed.

Each change in the code is called a mutant, and it results in an altered version of the program, called a mutation. Some types of mutation are:

Change conditionals Boundary Mutator.

Original conditional	Mutated conditional
<	<=
<=	<
>	>=
>=	>

Change mathematical operators.
Return null instead of Object value.
And many other types. Check this documentation for all available.

Mutations usually react as follows:

Killed: This means the mutant has been killed and therefore the part of the code that has been tested is properly covered.
Survived: This means the mutant has survived, and the added or changed functionality is not properly covered by tests.
Infinite loop/runtime error: This usually means that the mutation is something that could not happen in this scenario.

PITest framework is a JVM-based mutation testing tool with high performance and easy to use. I do not think this tool has competitors who have all of their features.

Getting started: Step by step with PITest 1.4.5 (2019 released version)

First, we will see how the jacoco code coverage is faulty.

Create a Demo App

1 - Go to https://start.spring.io/ and create a simple demo app (without site dependencies).

2 - Edit the pom.xml file, add Jacoco and maven plugins:



            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19.1</version>
            </plugin>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>report</id>
                        <phase>prepare-package</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

3 - Still in pom.xml file, add the unit testing dependencies.



        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <version>3.9.0</version>
            <scope>test</scope>
        </dependency>

4 - Create a simple service to verify whether a provided input number is between 0 and 100.

5 - Create a test class (without Asserts) like the one below:

Running the Demo App

Run mvn clean install in the root directory.

In this step, we can notice that our code is fully covered by unit tests. Open the jacoco report in target/site/jacoco/index.html.

Both line and branch coverage reports 100% unit tests coverage, but nothing is being tested really!

Adding the PITest plugin

We can limit code mutation and test runs by using targetClasses and targetTests.

And avoidCallsTo to keep specified line codes from being mutated. This improves the mutation time.



             <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>1.4.5</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals>
                            <goal>mutationCoverage</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <targetClasses>
                        <param>com.example.demo.service*</param>
                    </targetClasses>
                    <targetTests>
                        <param>com.example.demo.service*</param>
                    </targetTests>
                    <avoidCallsTo>
                        <avoidCallsTo>java.util.logging</avoidCallsTo>
                        <avoidCallsTo>org.apache.log4j</avoidCallsTo>
                        <avoidCallsTo>org.slf4j</avoidCallsTo>
                        <avoidCallsTo>org.apache.commons.logging</avoidCallsTo>
                    </avoidCallsTo>
                </configuration>
            </plugin>

Run the Demo App with PITest

Run mvn clean install in the root directory and look at the PITest report in /target/pit-reports/<date>/index.html.
Here we can notice the line coverage is still 100% but a new coverage has been introduced: Mutation Coverage.

Adding real tests with assertions

We can add asserts like this:

Run mvn clean install and check the PITest report again.

PITest executed tests after mutating our original source code and discovered some mutations are not handled by unit tests so we need to fix that.

To do so, we should cover cases including limit test case which means when the provided value is either 0 and 100.

Following are the test cases to cover mutation testing:



 @Test
  public void hundredReturnsTrue() {
    assertThat(cut.isValid(100)).isTrue();
  }

  @Test
  public void zeroReturnsFalse() {
    assertThat(cut.isValid(0)).isFalse();
  }

Running again the PITest mutation coverage command and looking at its report, we can now notice both line and mutation coverage look 100% good.

Bonus

We can use the property mutationThreshold to define a percentage of mutation at which the build will fail in case this percentage is bellow the threshold.

Performance of PITest in a real scenario

Running PITest in a small project (6300 lines of code) results in:



PIT >> INFO : MINION : 3:56:19 PM PIT >> INFO : Checking environment    
PIT >> INFO : MINION : 3:56:20 PM PIT >> INFO : Found  254 tests
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 5 seconds
> build mutation tests : < 1 second
> run mutation analysis : 2 minutes and 15 seconds
--------------------------------------------------------------------------------
> Total  : 2 minutes and 21 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 733 mutations Killed 690 (94%)
>> Ran 1158 tests (1.58 tests per mutation)

For this project, PITest showed that it generated a total of 733 mutations and of this total only 43 survived, resulting in 94% of mutation coverage.

Mutation testing can be a heavy process, but from my experience, by reaching 85% of mutation coverage, my team felt safe enough to make releases without manually testing the product. (That's cool!)

Conclusion

Note that code coverage is still an important metric, but sometimes it is not enough to guarantee a well-tested code. Mutation testing is a good additional technique to make unit tests better.