Killing mutants to improve your tests
Published by Manuel Rivero on 14/05/2019
One of the pieces we have developed to achieved that goal is reffects-store. Using this store, React components can be subscribed to given reselect’s selectors, so that they only render when the values in the application state tracked by the selectors change.
After we finished writing the code for the store, we decided to use mutation testing to evaluate the quality of our tests. Mutation testing is a technique in which, you introduce bugs, (mutations), into your production code, and then run your tests for each mutation. If your tests fail, it’s ok, the mutation was “killed”, that means that they were able to defend you against the regression caused by the mutation. If they don’t, it means your tests are not defending you against that regression. The higher the percentage of mutations killed, the more effective your tests are.
There are tools that do this automatically, stryker is one of them. When you run stryker, it will create many mutant versions of your production code, and run your tests for each mutant (that’s how mutations are called in stryker’s’ documentation) version of the code. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. Let’s have a look at the the result of runnning stryker against reffects-store’s code:
Notice how stryker shows the details of every mutation that survived our tests, and look at the summary the it produces at the end of the process.
All the surviving mutants were produced by mutations to the
store.js file. Having a closer look to the mutations in stryker’s output we found that the functions with mutant code were
After a quick check of their tests, it was esay to find out why
unsubscribeAllListeners was having surviving mutants. Since it was a function we used only in tests for cleaning the state after each test case was run, we had forgotten to test it.
However, finding out why
unsubscribeListener mutants were surviving took us a bit more time and thinking.
Let’s have a look at the tests that were exercising the code used to subscribe and unsubscribe listeners of state changes:
If we examine the mutations and the tests, we can see that the tests for
unsubscribeListener are not good enough. They are throwing an exception from the subscribed function we unsubscribe, so that if the
unsubscribeListener function doesn’t work and that function is called the test fails. Unfortunately, the test passes also if that function is never called for any reason. In fact, most of the surviving mutants that stryker found above have are variations on that idea.
A better way to test
unsubscribeListener is using spies to verify that subscribed functions are called and unsubscribed functions are not (this version of the tests includes also a test for
After this change, when we run stryker we got the following output:
No mutants survived!! This means this new version of the tests is more reliable and will protect us better from regressions than the initial version.
Mutation testing is a great tool to know if you can trust your tests. This is event more true when working with legacy code.
Originally published in Manuel Rivero's blog.