Mutation test introduces a way to find faults in our software components, verifying that our tests are able to detect unexpected behaviour. If they are able to find out the bugs caused by the mutations, we can affirm that our tests have good quality. However if defective tests are present, we should bear in mind that maybe a refactor of our tests will be needed to ensure good quality software.

PIT, is a mutation testing tool for JAVA, which is applicable in real world projects. PIT, is fast, robust and well integrated with other frameworks or tools, as it can be used vía Maven, Ant or terminal.

Thanks to this mutation tool, we will be able to apply a huge quantity of mutation of our code.

Is it not code coverage enough?.

TL;DR: No.

Nowadays it is well known by everybody the fact that software testing aims to check the expected behaviour of the software, making us to be able to detect bugs. One of the most important metrics in the industry of software development is code coverage.

But, is this a proper approach?. Should we care about something else? Why is not that enough?.

We can answer all theses questions with only one answer: because although our test is passing and we assume that our program works properly, having our lines covered does not mean that our pieces are well tested.

That's why mutation testing can be our friend - this technique will introduce defects vía small code modifications, and thanks to that and if our tests are consistent enough (so tests will fails if mutants are present), we can increase how well our software is tested.

👉 Code Coverage is a measurement of the percentage of code lines executed during the test suite.

Mutation testing.

Before jumping to the practical example, let's review the basic concepts of mutation testing.

• Mutant operators: The ones who transform the syntax of the program, for instance, the expression a + b will be changed to a - b or a operator like < can be changed to >.
• Mutants: Used to measure how good our tests are by observing and comparing the runtime behaviour of the non-mutated and mutated programs.
• Mutations: We can find two types of mutations:
  • Killed: Implies that there is at least one failing test as result of the mutation ✅.
  • Survived: It means that zero tests detected the mutation and it must be improved ❌

Mutation testing example.

Today we're presenting a simple example of how can we apply PIT in a project, in this case, we are going to use a PoC that was developed some time ago. Tests weren't perfect and coverage was not the best, and that's why it can be a good example of how we can improve our project testing quality by relying on this framework.

As we have spotted before, PIT, is available for Maven, so getting started is as easy as adding the plugin to our pom.xml , inside the <build> tag.

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.6.4</version>
    <dependencies>
        <dependency>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-junit5-plugin</artifactId>
            <version>0.14</version>
        </dependency>
    </dependencies>
</plugin>

Now, we are able to run our test using the following instruction mvn --fail-at-end clean test pitest:mutationCoverage pitest:report-aggregate-module and a report will be generated, where we will be able to inspect the mutants, the killed and the survived.

This report will be available in the target folder, under the pit-report a index.html will be present:

Now we can open it in our browser and investigate the possible issues present in our codebase according to PIT.

As we can see, mutation coverage reports a 24% of mutation coverage, 27% of line coverage (based on our unit test coverage report). These is the global report for our project, which for simplicity we have configured PIT to exclude configuration classes, domain entities and repositories, leaving our services as the key for the success of the testing sources.

To be more concise and to exemplify how can we deal with PIT, we are going to focus on the com.bnd.io.discounts.service.impl package, and the CouponServiceImpl.java class.

Report for service package. ☝️

As we can see in this first report, the mutation coverage for the class of interest is 0%. But which are the mutants we have to solve?. We can have more details if we navigate to the class, and check what PIT points out.

CouponServiceImpl.java class ☝️

Mutations KILLED & SURVIVED ☝️

Wow, there's a lot of information here... let's start step by step. So in our service class we have some methods, this won't be extrange for anyone, repository calls, business logics, etc. And then after we executed our mutation test platform, it reports that we have some pitfalls, some methods are not even tested and the mutants, have survived.

Also a list of the active mutations is present, so we can know what is being applied in our test suite.

Killing the mutants...

Having concerned the above points, now, it's time to kill the mutants. So, let's go for it! 🚀

To be begin with we are going to focus on the mutations annotated as SURVIVED, we will deal with the NO_COVERAGE tests later. So solve the problem with the findAll we need to change our unit test CouponServiceImpl::findAll method, or add a new one:

@Test
void testFindAllShouldNotReturnNull() {
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
    when(this.couponRepository.findAll(any(Pageable.class))).thenReturn(new PageImpl<>(List.of(coupon)));
        final Page<Coupon>result = couponServiceImpl.findAll(Pageable.unpaged());
        assertNotNull(result);
}

After running PIT, we added the previous test to kill that survived mutant. First mutant killed! 🤘

Let's add some more tests to have everything in place, for these first three tests, which mutants survived at the beginning:

We added these two test cases to achieve the current result:

@Test
void testFindOneShouldNotReturnEmpty() {
    final Coupon coupon = easyRandom.nextObject(Coupon.class);
    when(this.couponRepository.findById(any())).thenReturn(Optional.of(coupon));
        final Optional<Coupon>result = couponServiceImpl.findOne(1L);
    assertTrue(result.isPresent());
}

@Test
void testDelete() {
    couponServiceImpl.delete(1L);
        final Optional<Coupon>optionalCoupon = this.couponServiceImpl.findOne(1L);
    assertTrue(optionalCoupon.isEmpty());
    verify(couponRepository).deleteById(anyLong());
}

We still have to deal with some tests, as you can see, because we don't have even coverage for them, let's go for that!.

He have to bear in mind that we don't want to have any mutant!. Moreover it's important to focus on our tests verifications, this way, PIT mutants won't be able to survive, if the framework uses mutants of type removed call to → survived.

So finally, our test cases will remain as follow in order to kill all mutants present at the beginning.

@ExtendWith(SpringExtension.class)
class CouponServiceImplTest {

    public static final String COUPON_CODE = "CODE";
    @Mock
    private CouponRepository couponRepository;

    @InjectMocks
    private CouponServiceImpl couponServiceImpl;

    private EasyRandom easyRandom;

    @BeforeEach
    public void setup() {
        easyRandom = new EasyRandom();
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
        when(this.couponRepository.findByCouponCodeAndActiveIsTrue(COUPON_CODE))
                .thenReturn(Optional.of(coupon));
        when(this.couponRepository.findAllByDiscountType_DiscountTypeCodeEqualsIgnoreCaseAndActiveIsTrue(COUPON_CODE))
                .thenReturn(List.of(coupon));
        when(this.couponRepository.findByCouponCode(COUPON_CODE))
                .thenReturn(Optional.of(coupon));
    }

    @AfterEach
    void tearDown() {
        verifyNoMoreInteractions(couponRepository);
    }

    @Test
    void testFindAll() {
        final Page<Coupon> result = couponServiceImpl.findAll(Pageable.unpaged());
        assertNull(result);
        verify(couponRepository).findAll(any(Pageable.class));
    }

    @Test
    void testFindAllShouldNotReturnNull() {
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
        when(this.couponRepository.findAll(any(Pageable.class))).thenReturn(new PageImpl<>(List.of(coupon)));
        final Page<Coupon> result = couponServiceImpl.findAll(Pageable.unpaged());
        assertNotNull(result);
        verify(couponRepository).findAll(any(Pageable.class));
    }

    @Test
    void testFindOne() {
        final Optional<Coupon> result = couponServiceImpl.findOne(1L);
        assertEquals(Optional.empty(), result);
        verify(this.couponRepository).findById(anyLong());
    }

    @Test
    void testFindOneShouldNotReturnEmpty() {
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
        when(this.couponRepository.findById(any())).thenReturn(Optional.of(coupon));
        final Optional<Coupon> result = couponServiceImpl.findOne(1L);
        assertTrue(result.isPresent());
        verify(this.couponRepository).findById(anyLong());
    }

    @Test
    void testDelete() {
        couponServiceImpl.delete(1L);
        final Optional<Coupon> optionalCoupon = this.couponServiceImpl.findOne(1L);
        assertTrue(optionalCoupon.isEmpty());
        verify(couponRepository).deleteById(anyLong());
        verify(couponRepository).findById(anyLong());
    }

    @Test
    void saveShouldWorkProperly() {
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
        when(this.couponRepository.save(any(Coupon.class))).thenReturn(coupon);

        final Coupon savedCoupon = this.couponServiceImpl.save(coupon);
        assertNotNull(savedCoupon);
        verify(this.couponRepository).save(any(Coupon.class));
    }

    @Test
    void saveShouldThrowException() {
        final DataIntegrityViolationException exception = new DataIntegrityViolationException("Entity exists in database");
        final Coupon coupon = easyRandom.nextObject(Coupon.class);
        when(this.couponRepository.save(any(Coupon.class))).thenThrow(exception);

        DataIntegrityViolationException assertThrows = assertThrows(
                DataIntegrityViolationException.class,
                () -> this.couponServiceImpl.save(coupon)
        );

        verify(this.couponRepository).save(any(Coupon.class));
        assertNotNull(assertThrows);
        assertNotNull(assertThrows.getMessage());
        assertTrue(assertThrows.getMessage().contains("Entity exists in database"));
    }

    @Test
    void findByCouponCodeAndActiveIsTrueShouldFindProperCoupon() {
        final Optional<Coupon> optionalCoupon = this.couponServiceImpl.findByCouponCodeAndActiveIsTrue(COUPON_CODE);
        assertTrue(optionalCoupon.isPresent());
        verify(this.couponRepository).findByCouponCodeAndActiveIsTrue(anyString());
    }

    @Test
    void findByCouponCodeAndActiveIsTrueShouldNotFindAnyCoupon() {
        final Optional<Coupon> optionalCoupon = this.couponServiceImpl.findByCouponCodeAndActiveIsTrue("ANOTHER_CODE");
        assertTrue(optionalCoupon.isEmpty());
        verify(this.couponRepository).findByCouponCodeAndActiveIsTrue(anyString());
    }

    @Test
    void findAllByDiscountTypeCodeAndActiveIsTrueShouldWorkProperly() {
        final List<Coupon> coupons = this.couponServiceImpl.findAllByDiscountTypeCodeAndActiveIsTrue(COUPON_CODE);
        assertFalse(coupons.isEmpty());
        verify(this.couponRepository).findAllByDiscountType_DiscountTypeCodeEqualsIgnoreCaseAndActiveIsTrue(anyString());
    }

    @Test
    void findByCouponCodeShouldWorkProperly() {
        final Optional<Coupon> optionalCoupon = this.couponServiceImpl.findByCouponCode(COUPON_CODE);
        assertTrue(optionalCoupon.isPresent());
        verify(this.couponRepository).findByCouponCode(anyString());
    }
}

Excerpt of the full test suite for CouponServiceImplTest

What have we done in the previous test suite to achieve the a report with full succes?. We added verifications, and assertions to ensure that behaviour for our tests is correct. This way, we solved the different kind of mutants present in the report.

And now this is how our CouponServiceImpl report looks like:

Improved PIT report.

As you can see in the new report, after killing the mutants and dealing with the NO_COVERAGE tests, we have increased our package report statics.

Conclusions.

Having concerned all the above points, we can notice the flaws of only relying in the code coverage as metric. Evergreen tests can be present, which doesn't means that we are testing properly our software components. Thanks to the evolution of mutation tests, and within the combination of the code coverage metric, we can assure good quality software.

Source code for this example, can be found here.

Sources and references:

• https://mkyong.com/maven/maven-pitest-mutation-testing-example/
• https://www.baeldung.com/java-mutation-testing-with-pitest
• http://pitest.org/quickstart/basic_concepts/
• https://pitest.org/quickstart/mutators/
• https://pedrorijo.com/blog/intro-mutation/
• https://ulir.ul.ie/bitstream/handle/10344/5518/Cole_2009_DEMO.pdf;sequence=2