100 Percent Unit Test Coverage Is Not Enough

Code coverage tools trace the execution of your code and provide metrics about that execution. One of the most common measures is statement coverage. Statement coverage gives a percentage of the statements executed over the total number of statements in the program. Many organizations set goals for unit test coverage, with a common target being 80 percent statement coverage.

Developers pride themselves on getting to 100 percent unit test coverage, and people on the project teams associate this with having high-quality code. But even though the tests execute every line of code and we then call the code “fully tested,” this can be misleading.

One hundred percent unit test coverage does not mean we had good tests, or even that the tests are complete. The tests could be missing important data and only testing with data that succeeds, failing to test data that causes failures. One hundred percent unit test coverage doesn’t say anything about missing code, missing error handling, or missing requirements.

Tests also might not actually check the functionality of the code. Merely executing the code without checking its functionality still counts in the coverage metrics.

The following examples illustrate what can go wrong with relying on unit test coverage alone. The code is Python, and tests are written with the unittest module. Hopefully these examples inspire testers to review unit tests, improve these tests, or supplement them with functional tests to fill in the gaps—even if test coverage is 100 percent.

Missing Test Cases

Consider the following function (only_correct_data), which takes several parameters, performs some math, and returns the result. It would only take one unit test to achieve 100 percent code coverage, because this function only has one line of code:

def only_correct_data (a, b, c) :

    return (a / (b - c))

def test_only_correct_data(self):
   #only tests with data that leads to correct results

    self.assertEqual( only_correct_data(1,2,3) , -1)
    self.assertEqual( only_correct_data(2,3,1) , 1)
    self.assertEqual( only_correct_data(0,2,3) , 0)

The test coverage is 100 percent, with three test cases that cover a negative result, a positive result, and a zero result. What is missing is the test case that would create a divide by zero error. Using the call only_correct_data(2, 2, 2) would result in the divide by zero exception. Finding missing tests cases for the developer is a great way to improve the unit tests.

Another benefit of reviewing these unit tests and making sure they are thorough is that the tester can concentrate on the overall system quality, the user experience, and acceptability to the customer. Knowing that the math is correct and verified by unit tests allows the tester to concentrate on these other factors related to quality.

Missing Functionality 

The unit tests typically verify that the developer’s intent is implemented correctly, but even 100 percent code coverage does not mean the requirements are fully met. 

In this example, there is a class, called “Dog,” that returns a “bark” based on the size of the dog. 

class Dog(object):

    def __init__(self, size):
       if size == "Large" :
           self.bark = "Woof!"
       if size == "Small" :
           self.bark = "Bow Wow"

def test_missing(self):
   #missing a test for medium dogs    

    l = Dog("Large")    
    self.assertEqual( l.bark, "Woof!" )    
    s = Dog("Small")
   self.assertEqual( s.bark, "Bow Wow" )

The Dog class is missing the medium-sized dog that barks “Ruff.” The code is missing a requirement, and thanks to additional testing at this stage, the bug was found well before acceptance testing, when it takes much less effort to fix. 

This example shows that having 100 percent code coverage doesn’t tell you anything about code that is missing. The unit tests will check that the code is working as the developer intended the code to work, but not necessarily that the code meets customer requirements.

Bad or Incorrect Tests

Having the code 100 percent executed during testing also does not mean the tests are actually good tests, or even test anything at all.

def rev_string( in_string) :
   return in_string 

def test_bad_test(self):
   #does not actually check if code is correct

    self.assertIsNotNone(rev_string("Test String"))

In this example, the function should reverse the string, but it doesn’t. Even though the unit test has 100 percent coverage, it never actually checks if the string is reversed. It’s only checking that the return value is not “None.” The test case could just execute the code without actually testing anything and still achieve 100 percent code coverage. The tester should double-check the quality of these tests or create tests that actually verify the functionality.

    do_nothing = rev_string("Test String")

This line will execute rev_string but not actually test it, still achieving 100 percent coverage. In these cases the tester could help the developer think through the proper ways to test the functionality to fill in the testing gap.

Exception Handling

This example shows a very common situation where the code has some error checking but is not fully tested. The code coverage for all the code created for this article is 98 percent, but the one missing test here is an important one.

def missing_important( a, b) :

    if a is None :
       #handle exception
       a = 1 / 0  #intention bug
   return (a+b)

def test_incomplete(self):
   # exception handling not tested        
   self.assertEqual (missing_important(1,2), 3)

Unit testing is a fantastic way to test exception handling because the developer has full control of the context and can inject the conditions necessary to test the errors. The tester, working at system level or through an API, might not be able to create the condition where, in this example, the variable “a” has the value of “None.”

Next Steps for Testing 

These examples are simple in order to demonstrate these points, and of course in practice, unit tests are valuable. But a high percentage of unit test coverage does not automatically equal high code quality; the tests need to be effective, as well.

Testers are still needed to review the unit tests to make sure they are the best tests possible, to improve or supplement unit tests with tests of their own, to keep the customer requirements perspective in mind when reviewing the unit tests, and to identify areas that are covered by unit testing that can help optimize the integration and system tests.

If you have other suggestions, please mention them in the comments section below.