Testing Nirvana - 100% Code Coverage
22 May, 2015
In my last post I introduced testing a PowerShell scripts, now lets take that a little deeper and improve on the testing.
“Improve on the testing” you say, “how is that possible?”
When you are testing a script, or function or any piece of code, you want to be sure not only to test the main path, the normal case, but all the different execution paths. Every if statement takes the execution through a different sequence and if there are some without test cases then you might have issues (that’s fancy for bugs) that are going to bite you later on.
There are a couple of different ways to go about doing this.
Test Driven Development
One option is to write your tests before you write your code (something called Test Driven Development or TDD). If you’re following this style of development then you’re creating a test before you even add a parameter or think about processing one kind of data differently. This option is also very good for the testability (that’s fancy for how easy it is to test) of your implementation because it forces you to find a way to test before you even start writing, rather than struggling to find a way to test something you’ve already written, and possibly having to go back and rewrite it to make it testable.
Code Coverage
Another option, which is particularly useful if you are writing your tests after the fact, is to use the Code Coverage feature of Pester. What is Code Coverage? Code coverage means what percentage of your code is covered by tests. A particular line of code is considered covered if it is executed by one or more test cases. Having 100% code coverage should be your ideal goal although sometimes it’s not possible. If you really aren’t able to achieve 100% code coverage you should be sure it’s for the right reasons and seriously consider if it may be because the implementation isn’t testable rather than the test case not being testable.
Dead Code
Code coverage, when you are using it after the fact, can also help discover dead code. I once read an article that software developer was a terrible term because it made it sound like there was a definate end. Instead, the article proposed the term software gardener because software can be very much like a living thing that grows but also needs pruning. When you have parts of your application that are no longer used, the right thing &trad; to do isn’t to leave them there but to remove them. Besides, if you ever needed them again, you could just checkout an older copy of your code from your SCM (you are using some kind of Revision Control tool like Git aren’t you?).
So How Did I Do?
A quick check of the code coverage of the tests from my last example shows less than 100%. Let’s dig into why.
PS> Invoke-Pester .\Get-PhoneNumberClass.Tests.ps1 -CodeCoverage .\Get-PhoneNumberClass.ps1
Code coverage report:
Covered 67.97 % of 128 analyzed commands in 1 file.
Missed commands:
File Function Line Command
---- -------- ---- -------
Get-PhoneNumberClass.ps1 ClassifySlow 156 Write-Verbose "Classifying $number (slow)"
Get-PhoneNumberClass.ps1 ClassifySlow 157 $class = "Ordinary"
Get-PhoneNumberClass.ps1 ClassifySlow 158 $reason = ""
Get-PhoneNumberClass.ps1 ClassifySlow 159 @("Gold","Silver","Bronze")
Get-PhoneNumberClass.ps1 ClassifySlow 159 "Gold","Silver","Bronze"
Get-PhoneNumberClass.ps1 ClassifySlow 160 $CLASSES[$c].GetEnumerator()
Get-PhoneNumberClass.ps1 ClassifySlow 160 foreach {...
Get-PhoneNumberClass.ps1 ClassifySlow 161 if($number -match $_.Value) {...
Get-PhoneNumberClass.ps1 ClassifySlow 162 $class = $c
Get-PhoneNumberClass.ps1 ClassifySlow 163 $reason = $_.Key
Get-PhoneNumberClass.ps1 ClassifySlow 168 @{Number = $number; Class = $class; Reason = $reason}
Get-PhoneNumberClass.ps1 ClassifySlow 168 Number = $number
Get-PhoneNumberClass.ps1 ClassifySlow 168 Class = $class
Get-PhoneNumberClass.ps1 ClassifySlow 168 Reason = $reason
Get-PhoneNumberClass.ps1 196 $numbers = 0..$RunSize | %{ Get-Random -Minimum 1991000000 -Maximum 99999...
Get-PhoneNumberClass.ps1 196 $numbers = 0..$RunSize | %{ Get-Random -Minimum 1991000000 -Maximum 99999...
Get-PhoneNumberClass.ps1 196 Get-Random -Minimum 1991000000 -Maximum 9999999999
Get-PhoneNumberClass.ps1 198 Measure-Command { foreach($n in $numbers) { ClassifyFast $n | out-null } }
Get-PhoneNumberClass.ps1 198 $numbers
Get-PhoneNumberClass.ps1 198 ClassifyFast $n
Get-PhoneNumberClass.ps1 198 out-null
Get-PhoneNumberClass.ps1 199 select @{n='Name';e={'Fast'}},TotalMilliseconds
Get-PhoneNumberClass.ps1 199 n = 'Name'
Get-PhoneNumberClass.ps1 199 e = {'Fast'}
Get-PhoneNumberClass.ps1 199 'Fast'
Get-PhoneNumberClass.ps1 201 Measure-Command { foreach($n in $numbers) { ClassifySlow $n | out-null } }
Get-PhoneNumberClass.ps1 201 $numbers
Get-PhoneNumberClass.ps1 201 ClassifySlow $n
Get-PhoneNumberClass.ps1 201 out-null
Get-PhoneNumberClass.ps1 202 select @{n='Name';e={'Slow'}},TotalMilliseconds
Get-PhoneNumberClass.ps1 202 n = 'Name'
Get-PhoneNumberClass.ps1 202 e = {'Slow'}
Get-PhoneNumberClass.ps1 202 'Slow'
Get-PhoneNumberClass.ps1 204 Measure-Command { foreach($n in $numbers) { ClassifySlowOptimized $n | ou...
Get-PhoneNumberClass.ps1 204 $numbers
Get-PhoneNumberClass.ps1 204 ClassifySlowOptimized $n
Get-PhoneNumberClass.ps1 204 out-null
Get-PhoneNumberClass.ps1 205 select @{n='Name';e={'Slow Optimized'}},TotalMilliseconds
Get-PhoneNumberClass.ps1 205 n = 'Name'
Get-PhoneNumberClass.ps1 205 e = {'Slow Optimized'}
Get-PhoneNumberClass.ps1 205 'Slow Optimized'
Right away I can see that this points to two sections in my script; section 1 is lines 156 - 168 (the function ClassifySlow) and section 2 is lines 196-205 (this is the part of PROCESS dealing with the -Test flag).
How to remedy this? I’ve got a couple options here. I could remove both sections, actually I could go further and remove the -Slow option as well. Since this would introduce an incompatible backwards change (you couldn’t call the script with -Test and get the behavior you currently expected) that would mean bumping this to version 2.0.0. If you’re unfamilar with versioning, I strongly recommend reading Semantic Versioning and following it as a standard. If you are familar with SemVer then you’ll have recognized this already. Another option would be to add test cases that covered both sections. And a third option would be a little of both. That’s actually the option I’m going to go for.
For section 1 I’m just going to remove it since that version of the Classify function was one I did purely to compare the performance of different approaches and the only place it’s used currently is by the -Test section. For section 2, I’ll add a test case to cover this function:
Context "When a performance test is specified" {
$ExpectedAlgorithms = @("Fast", "Slow")
It "returns a list of algorithms and their total milliseconds" {
[array]$results = & $cmd -Test
$results.Length | Should Be $ExpectedAlgorithms.Length
foreach($algo in $ExpectedAlgorithms) {
$r = $results | where Name -eq $algo
# in the latest Pester this is the same as
# $r | Should Exist
$r | Should Not Be $null
$r.TotalMilliseconds | Should Match "^\d+\.\d+$"
}
}
}
From a semantic versioning perspective, given that the output of -Test is an array of hashes with the algorithm name and a time taken, I interpret this to be a patch level change; a backwards compatible bug fix. If the output of -Test had been an object with each algorithm as a property like this:
PS> [pscustomobject]@{Fast = 0.234; Slow = 0.345; SlowOptimized = 0.456}
Fast Slow SlowOptimized
---- ---- -------------
0.234 0.345 0.456
Then that would have been a backwards incompatible change.
What’s The Coverage Now?
So with those changes, what’s the coverage look like?
PS> Invoke-Pester .\Get-PhoneNumberClass.Tests.ps1 -CodeCoverage .\Get-PhoneNumberClass.ps1
Code coverage report:
Covered 100.00 % of 106 analyzed commands in 1 file.
Now that’s what I want to see!
