Code coverage is a useful way of checking that you are fully testing your code. When you run unit tests, one of the goals is to check that the results match your expectations. But how do you know if you have all the cases covered? Your tests may be passing, but if you only checked 50% of the possible cases, how can you be confident in it?
Early one morning, a programmer asked the great master:
“I am ready to write some unit tests. What code coverage should I aim for?”
The great master replied:
“Don’t worry about coverage, just write some good tests.”
The programmer smiled, bowed, and left.
A small example
As an example, the class below is for a simple static function that checks if a number if a prime number.
class PrimeCheck
{
/**
* Given a number, returns true if it is a prime number or false if it is not.
*
* @param $num int The number to check for primality.
*/
public static function isPrime($num)
{
$num = (int) $num;
// 1 is not prime.
if ($num == 1) {
return false;
}
// 2 is prime.
if ($num == 2) {
return true;
}
// Divisble by two then not prime.
if ($num % 2 == 0) {
return false;
}
// Test odd numbers up to the sqrt (otherwise double the work!)
for ($i = 3; $i <= ceil(sqrt($num)); $i = $i + 2) {
if ($num % $i == 0) {
return false;
}
}
// If not returned, must be prime.
return true;
}
}
Our unit test currently looks like the below (because we didn’t follow TDD when writing our code!):
class PrimeCheckTest extends \PHPUnit_Framework_TestCase
{
/**
* Ensure 4 is not regarded as a prime number.
*/
public function testFourNotPrime()
{
$this->assertFalse(PrimeCheck::isPrime(4));
}
/**
* Ensure 2 is a prime number.
*/
public function testTwoIsPrime()
{
$this->assertTrue(PrimeCheck::isPrime(2));
}
}
Just how much of our class is it testing? PHPUnit lets us find out this information very easily with the coverage option. From a terminal, run the following commands:
$ phpunit --coverage-html ./coverage/ ./PrimeCheckTest.phpThe coverage report
In the coverage folder you’ll now find several HTML files (you can generate a shorter text summary with --coverage-text.) Opening these in a browser will show the following:
By clicking on the tests folder and finding the PrimeCheck class file you can find a summary of what your test did:
This tells us only 50% of the code was actually run by our test! Going further and clicking on the class file name lets you see exactly what lines of code have been called. Green indicates that the test did involve that line, whilst red shows it was never called and white means it isn’t really relevant.
This analysis shows we’ve not actually tested to check the special case of one, and we also haven’t tested the loop at the bottom either.
Making it better
Later that day, a second programmer asked the same question.
The great master pointed at a pot of boiling water and said:
“How many grains of rice should put in that pot?”
The programmer, looking puzzled, replied:
“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”
“Exactly,” said the great master.
The second programmer smiled, bowed, and left.
Based on this insight, our updated test now looks like:
class PrimeCheckTest extends \PHPUnit_Framework_TestCase
{
/**
* Ensure 9 is not regarded as a prime number.
*/
public function testNineNotPrime()
{
$this->assertFalse(PrimeCheck::isPrime(9));
}
/**
* Ensure 7 is not regarded as a prime number.
*/
public function testSevenIsPrime()
{
$this->assertTrue(PrimeCheck::isPrime(7));
}
/**
* Ensure 4 is not regarded as a prime number.
*/
public function testFourNotPrime()
{
$this->assertFalse(PrimeCheck::isPrime(4));
}
/**
* Ensure 2 is a prime number.
*/
public function testTwoIsPrime()
{
$this->assertTrue(PrimeCheck::isPrime(2));
}
}
Regenerating the coverage report and opening it, you should find that we’ve achieved nearly 100% coverage! It’s an exercise for the reader to use this to identify the missing tests.
Toward the end of the day, a third programmer came and asked the same question about code coverage.
“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.
The third programmer smiled, bowed, and left.
Interpreting the results
Having 100% coverage does not mean that the implementation is complete or correct in all cases at all though. What about a negative?
In fact, code coverage can only tell you what you have not tested, not what you have. The figure you get from the report cannot tell you anything about the “importance” of the code being tested or the quality of the tests written.
It is a useful tool to check just how much code has been tested and helping you come closer to proving the absence of certain types of bugs.
After this last reply, a young apprentice approached the great master:
“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”
The great master stood up from his chair:
“Come get some fresh tea with me and let’s talk about it.”
After they filled their cups with smoking hot green tea, the great master began to answer:
“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”
“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”
“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”
The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.
“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”
The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.
Quotes from the Way of Testivus
Leave a Reply