Continuous improvement can help completely turnaround a staid IT organization into an innovation machine. But without effective test automation, that transformation is a non-starter. So says Gary Gruver, who has led DevOps journeys both at HP and Macy’s and now works as a consultant guiding other organizations through his firm, Practical Large Scale Agile.

Last month at IBM InterConnect, Gruver gave audience tons of great big-picture advice on leading DevOps from the top of the food chain. But perhaps one of the best takeaways when drilling down into the details was his emphasis on test-driven development.

“It needs to be reliable, it needs to be stable, it needs to be maintainable. i would say test automation is something that’s the most important if you’re going to transform your  organization,” Gruver said. “And it’s one of the things that’s most often done wrong. If you do it right it’s extremely valuable, if you do it wrong it’s going to be a maintenance nightmare.”

He provided real world lessons he learned from his stints at Macy’s and HP, explaining, for example, a situation at HP where unstable tests and unstable codes on trunk had kept the team from checking in code for four days. With a $2 billion business “breathing down my neck,” Gruver admitted he was getting antsy at that point and found out that the instability prompted them to add even more tests to the process.

“So we weren’t getting any better because no code was making it in to get fixed and what we really needed is a process that lets the good code in and keeps the bad code out,” explaining that his team eventually settled on gated commits. “That’s where if the code doesn’t pass the system tests it needs to for red build, then it doesn’t make it into SCM and doesn’t make it on to the trunk.”

Meanwhile, when he got to Macy’s he was told there were 1,000 automated tests ready for use. But when he started marrying those tests with the burgeoning continuous delivery team, they found not a single one worked reliably enough to depend upon. He recommended to the audience to take a look at the architecture philosophy’s put forth in Jeff Morgan’s Cucumber and Cheese, which helped in his work at Macy’s, where by the end of his stint the IT department was running over 5,000 automated tests several times a day, versus 1,300 every 10 days.

He explained that he’s a firm believer that quality assurance should be brought together with the architects to make sure the automation framework is sustainable for the long haul.

“Get your lead architect on the development side to engage with your lead on the quality assurance side and think about how to architect your framework for your test automation where its maintainable,” he said. “If not, you’re going to die under the weight of test automation.”

If done right, an organization will very well likely write more code for test than for running its product.

“And that’s not a bad thing,” he says. “You’ll find when you write good test automation it forces you to write better code.”