Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.
DeArdo is no stranger here at DevOps.com, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.
Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.
DevOps.com: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?
DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.
What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.
Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.
These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?
DevOps.com: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?
DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.
But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”
DevOps.com: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.
DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.
We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.
DevOps.com: What are some of the high-level challenges that you’ve had optimizing that last mile phase?
DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.
So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.
I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.
One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.
That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.
That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.
DevOps.com: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.
Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?
DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.
DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.
Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.
But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.
