The Risks and Limitations of Relying Exclusively on Code Coverage Percentage
Valuable and effective testing is concerned with other aspects, code coverage is a consequence of quality, planned and comprehensive testing focused on the observable behavior of the feature.
It's a common misconception among some developers that refactoring code, which results in a decline in the total number of lines, will inevitably lead to an automatic improvement in the quality of our tests and reports. However, this reasoning lacks rationale and offers no real benefits to code quality or testing effectiveness.
It is imperative to understand that code coverage is a relative metric and should not be interpreted in isolation. Reducing the number of lines of code does not necessarily guarantee more robust or more complete tests. Refactoring should be oriented towards improving the readability, efficiency and maintainability of the code, not just reducing the number of lines.
To illustrate how a myopic approach focused only on code coverage metrics can be misleading, consider the following example:
The above code can be refactored, but please note that nothing will change in the unit tests, i.e. the behavior will remain the same. Here's how the report generated with row percentage will be affected after refactoring:
BEFORE REFACTORING
AFTER REFACTORING
Refactoring the code:
The percentage number has increased, it was 46% but now we are at 80%, does this mean that we have an evolution or improvement in the testing scenarios? No!
We only reduce the lines of code, but the behaviors and the important ramifications are not covered by the tests! We did not write any other test scenarios, the number of lines covered changed only because after the refactoring the number of lines decreased!
If you look at the before and after images, the percentage of branch coverage (paths that the algorithm can take) remains the same even with refactoring. And this is one of the big problems when working with code coverage, many times if we act without thinking or analyzing carefully, we end up being fooled!
It can be very easy to manipulate coverage numbers. The more compact your code is, the better the test coverage metric will be because it only considers the raw line numbers.
This is why careful analysis is required by developers and software testers, again looking and watching over the quality of the written tests. What does this prove? The code coverage per line, i.e. this percentage generated in the report, is not related to quality.
Just because your code is 100% covered does not mean that your tests reflect quality. All that we have just seen further reinforces that coverage is just one tool to use in order for us to create effective and quality tests, it is a tool that helps us, but we cannot fully depend and rely on it without thoroughly analyzing all the inputs and outputs.
Furthermore, a test needs to have many important attributes to be considered a quality test, for example, it needs to be readable, organized, easy to understand, without dozens of asserts and mainly focused on testing behaviors of the functionality. So we just answered the question: Does code coverage indicate quality?
It is not indicative of quality! Valuable and effective tests are concerned in other respects, code coverage is a consequence of quality, planned and comprehensive tests focused on the observable behavior of the feature!
📍The dangers of relying completely on code coverage metrics
Focusing on quantitative metrics at the expense of quality: Too much emphasis on the percentage of code coverage can lead the team to focus only on quantitative metrics, disregarding important qualitative aspects of software quality. This can result in inefficient testing and a lack of identification of potential problems.
Neglecting important aspects of testing: By focusing only on code coverage, the team may neglect other important aspects of testing, such as the quality and relevance of test cases, which can result in poorly designed or irrelevant tests that do not contribute to the actual improvement of software quality.
False sense of security: Relying completely on the percentage of code coverage can lead to a false sense of security, making the team believe that the software is well tested and free of defects, even when there are quality problems in the tests or use cases that are not covered.
Ignore high risk or complexity areas: Code coverage does not take into account the complexity of the code and the interactions between different components of the system. This means that high-risk or complex areas may not receive proper attention, even if the coverage percentage is high.
Based on the above list of the dangers of relying only on coverage percentages, it is clear what problems we can encounter in software and even during development if we blindly rely on percentages and metrics. It is important to reinforce these risks with the programming team, software managers, and anyone else involved. After all, we want a product that is really reliable!
📍How can we use code coverage metrics to our advantage?
How can we use this tool to our advantage without falling into the pitfalls of line coverage? We can list some suggestions, based on what other experts share and also based on some experiences from projects that I have worked on:
Use this tool in areas of the code that are more complex and critical to the type of business you are working on.
If the team finds that a file has zero coverage or a very low percentage and number of branches that have not been covered, it helps to write test scenarios for behavior.
Reinforce the importance of testing code branches! Branches indicate the paths that an algorithm can take to accomplish its goal. So if your code contains many if/else structures, it is always worth looking at how the tests are built on top of these branches and if they exist.
Pay attention to the coverage report, but never assume that the tests were written with quality and cover all the behaviors of the functionality, just based on the report. As we have seen, 100% code coverage has no relation with test quality.
There's no point in establishing high percentages in the project and even in the files if the quality in writing tests and scenarios remains insufficient or inadequate.
Tools that generate coverage reports can show you the complexity of the cyclomatic code and this can help you indicate whether or not you should look into refactoring the class or methods. So this is a very positive factor. Always check to see if refactoring is possible to improve the readability of your code.
Code reviews and collaboration, code coverage metrics can provide useful information for code reviews, helping reviewers to focus on less tested areas and identify potential risks.
Carefully review the reports and percentages shown, never ignore other test scenarios!
📍 Conclusion
It is important to highlight the problems of completely relying on code coverage percentages as the only metric for evaluating the quality of tests and software in general. While code coverage is a valuable tool for identifying areas that need more attention in testing, it is critical to understand its limitations and not use it as the only indicator.
One of the main problems with relying exclusively on code coverage is the false sense of security that can be generated. A high percentage of coverage can lead the team to believe that the software is well tested and free of defects. We don't want to be deceived!
Another problem is the tendency for teams to focus only on increasing coverage percentages, neglecting other important aspects of testing, such as the quality and relevance of test cases. Badly designed or irrelevant tests may increase coverage, but they do not contribute to real improvement in software quality.
We have clearly seen that this metric can be manipulated. Stay alert! Avoid being surprised. Write tests that are really valuable and use this tool to your advantage!
I hope this post helped you, if you liked it, please share it with others! See you next post! 😉😄
Books that I consider essential for everyone ✅:
Effective Software Testing: A Developer's Guide - by Mauricio Aniche
Unit Testing Principles, Practices, and Patterns - by Vladimir Khorikov