I grew up during the end of the Cold War. During that time, I remember hearing that there were enough nuclear weapons to destroy the world 20 times over. That always seemed silly to me. Wasn’t once or twice plenty? Wasn’t it more important to have the fastest and most effective weapons rather than to simply have the most weapons? Before I could devise a satisfactory explanation, “Rocky IV” was released, successfully ending the Cold War.
Number of Automated Tests: The New Iron Curtain
Now, 30 years later, I’m beginning to notice another arms race taking place. At a recent DevOps conference, I saw speaker after speaker boasting about the number of automated tests their respective shops were running daily. Again, I was confused. Those numbers provide no real information. What’s more important than the quantity of tests you are executing is the efficacy of those tests and the speed of execution. Why did no presenter have those numbers? It felt like Cold War braggadocio all over again.
Why do I care? Shouldn’t I view the embracing of test automation as a good thing, regardless of the metric used to measure it? When creating a culture of automation, test counts do serve a purpose. First of all, It’s a simple metric. Just count the number of tests and dance in the self-exulting joy of your automation prowess. It also shows the power of automation compared to manual execution. So what’s the problem? The reason I care is this: Test-case count, while simple to gather and easy to understand, puts the focus on irrelevant data and ignores more critical information, similar to judging a military on number of weapons alone (yes, I’m forcing the analogy). Number of test cases executed daily isn’t just the wrong metric; it’s the wrong unit of measurement! Trying to determine a development shop’s test automation maturity by counting number of tests executed daily is like trying to determine how many gallons a good developer weighs. It’s just nonsensical data that doesn’t answer the important questions:
- What percentage of your code base is covered by your automated tests? If you knew that your 5,000 tests only covered 15 percent of your code, would you still stand on a stage at a conference and boast?
- How long do your tests take to run? If it takes all night to execute the suite, then you can kiss multiple releases a day goodbye.
Using Proper Metrics, or the Strategic Defense Initiative
So what are the automated metrics we need to worry about? I have a few favorites, in no particular order:
- Test execution time: Speed matters. A decrease in testing time allows for an increase in number of test iterations, an increase in features per release, an increase in release frequency, etc.
- Percentage of code covered: The tool you use to get this number doesn’t matter. The key thing is that you have a standard way to track this metric and whether you are trending upward. This trend gives you the next important metric …
- Automation progress: How fast is the discipline growing? When are you projected to hit the point where all tests that can be automated have been automated?
- Defect density: While this isn’t specific to automation, the trend in this metric helps show the benefit of automated testing. Bonus points for having a pre-automation baseline.
- Number of automated test executions: WHAT? Hey, sometimes you have to parade the military through Red Square to put on a show of strength. This isn’t a bad metric when used in conjunction with the others. But it is not enough by itself!
Keep on Rockin’ in the Free World
By focusing on the above metrics and not solely on test execution counts, you will ensure you are moving in the right direction in terms of test automation. Similar to how Gorbachev moved the Soviet Union in the right direction with his policies of perestroika and glasnost. OK, OK. The analogy is officially dead.
