Enterprises have been working through production delays and operational inefficiencies for decades. Don’t get sucked in too deeply to the ever-present mantra of “this time is different”. There is a lot of source material out there from people in a different kind of operations that have already been through the same wringer as DevOps presents. You just have to know where to look and remember in the end “it’s about what works.”
I was talking to Lori about my concept for this blog, and she sent me a link to a pretty good (dated, but relevant in light of DevOps) PDF that introduces quite a few of the concepts. But let me pull from personal experience to try and show how little uniqueness is enshrouded in DevOps.
Like DevOps… In the past.
I was hired into a utility company a long time ago to take them from a run-of-the-mill utility to a fully implemented Automated Meter Reading utility. No utility in the country had gone to 100% automated for meter readings for a variety of reasons, some technological, some fear of change, some organizational. For the organizational part, imagine how many meter readers it takes to read 350,000 meters… Now imagine reorganizing without all or nearly all of those jobs. And imagine they type of jobs – automated meter reading operations and high-tech meter repair – that would replace them. It was a tall order.
My role was strictly datacenter. I was given a mostly empty chunk of DC real estate, and told to build. When I walked in, there was a server for collecting the readings of the system that won the prototype phase, and some modems to read some meters that were already automated when the project was first conceived.
While the engineering manager, operations manager, and myself overcame our various technological issues, others were dealing with the organizational issues. When I joined the company there were less than one percent of meters being read electronically. When I left the project two years later we were at 80% and the groundwork was laid to reach 100%. The organizational changes had been run through, both in terms of staffing issues, and in terms of resource utilization in IT. The transformation took a long time because physical hardware had to be installed on each meter.
But reading errors rapidly became an exception. Cases where there was too much snow and the meter reader had to estimate? A thing of the past. Estimation was now reserved for exception cases, as normal operational procedure covered the vast majority of meter reads, even in blizzard conditions. And cost went down. While the initial investment was extremely large for a utility of that size, the ROI was solid, and rapidly that money was being recouped.
What did they have to do? They had to conquer the three issues DevOps faces. Automate complex, error prone operations, make organizational changes that will serve the long term interests of operations, drive disparate groups to work together and understand issues impacting the overall project, and constantly monitor the ROI of the overall project.
For that project, we were lucky in that there was more than buy-in from executive management, their was strategic priority. We got what we needed as long as we showed we were on track. DevOps is unlikely to achieve that level of support unless C-level execs can see the ROI and how it will help the organization going into the future. That’s the topic of my next blog, but luckily we won’t need that level of buy-in.
…Is DevOps Today.
Get a project manager, give them an operational process that is either done so frequently as to warrant special attention or is error prone enough to warrant special attention, and tell them to figure out the steps, who is involved when, and come up with a plan to automate it that can be achieved in a reasonable amount of time.
Then set guidelines for what you will measure to prove success, and make certain you start measuring them in the current state if you aren’t already. Measurements are useless without a baseline.
For an oft-repeated process, reduction in the time it takes to complete the process OR in the manhours required to complete the process is a good first measure. For error prone processes, measure rates of error.
It is that easy to get started. While the PM will require some time of employees, if you choose the project well, the PM will be the only one devoting significant amounts of time to it… And you will come out the other end having a story to talk about. Hopefully having a success story, but even if the results are not enough to warrant further investigation, you’ll have a solid reason why.
Are we increasingly hearing about nifty cool tools and whiz-bangs? Yes. And if you have the funds, go explore them. But always remember, first came DevOps, only then did tools get developed. Some will help you, none are strictly necessary, and DevOps rarely gets an overflowing budget.
So hop on in there. You have done this a zillion times in each of the given areas of IT – how many standardized VM images or automated spin-up scripts do you have laying around for example? The only difference here is that you are doing it across more of IT, and you’re being more formal about it.
And if the results of your first project show little improvement in your chosen measurements, you will have to look closely at the project, the PM, the organization, and the measurements. Each of these will lend itself to different solutions, it’s up to you to decide what to do for your company at that point.
But likely, if you chose the project well, you’ll be off and running with little risk and much knowledge gained.
And the PDF Lori sent me? It is about Business Process Management. And if you look closely at the steps, you will find that BPM was doing for the business pretty much exactly what we’re doing for ourselves now – reducing turnaround time, saving money, and improving customer (in our case the business user) experience.