When I took a job as Technology Evangelist for StackEngine, my wife’s first question was, “Why would someone pay to fly you around the country and talk about how you work?” My youngest son put a finer point on it, “Yeah Dad, we love you and we don’t listen when you talk about work!”
I tried to explain Lean principles. That went over like a lead balloon. I tried a more human approach in explaining John Willis’ Karojisatsu post and the rampant stress and depression in the operations industry. They felt sorry for those people, but still didn’t see 100’s of folks crowding in a room to hear about it.
How am I going to foster organizational change when my own family doesn’t care? I needed a way to communicate my ideas so anyone could approach them.
So, I started doing a single load of laundry each day.
LaundryOps
Really it was pretty obvious looking back. My wife typically did our laundry each Sunday. We could either stay at home most of the day, or we could put it off and stress about running out of clean underwear. It’s the perfect metaphor for the problems with large batches.
The following Sunday my wife noticed there was about a third of a load in the hamper. She said, “Hey, there’s no laundry to do today, let’s go to the water park this morning before it gets crowded.”
My wife was pleased. The kids were thrilled. I was thinking, “Hmmm, this continuous delivery model is interesting.” I was also thinking that single piece flow provides supporting precedent.
LaundryOps – Continuous Delivery
In their seminal book Continuous Delivery (CD), Humble and Farley stress that CD makes the working environment more humane.
Lean principles say that I should identify what variable I am trying to impact. This tenant is expressed as the measurement pillar from the CALMS acronym coined by Willis and Damon Edwards.
My goal was to maximize quality family time. It was not to conserve water or energy. We have done a great deal more hiking, swimming and other fun things because our weekend is full two days.
Laundry Ops – Reasoning About Change
So, if I practice continuous delivery, and do a load a day, what happens when I go on a trip and come back with 5 days worth of clothes?
Well, the first time I panicked! It was apparent that CD did not apply to special situations. I needed to break process. But, after considering the pile of clothes, I realized that doing a large load today and tomorrow would bring the system back to equilibrium the following day.
Adopting CD thinking for our laundry exposed a stunning amount of spare capacity in our “softener, detergent, laundry cycle” or SDLC for short. I did, however, recognize the inclination to regress to my comfort zone at the first sign of trouble in the new process.
I need to evolve my thinking about the batch size of work.
Often during the software delivery life cycle (huh, SDLC too) a developer/project manager/stakeholder thinks: “I will do this one more thing before deploying.” This is often because the current deploy process is highly manual, risky or time-consuming. Certainly none of those things were true of laundry.
I need to find ways to make deploys easier, safer and repeatable.
ClosetOps – Unexpected Consequences
Doing one load per day rather than many on Sunday led to a number of other interesting results that all apply to software delivery.
ClosetOps – Cash Flow
If our clothes are not in the hamper, washer or dryer, then where are they? Turns out our closets were burgeoning. At this point my wife said, “I wonder how much money is in there?” So we went through her closet with It’s Deductible and learned the answer was about $3500. Retail we estimate $8500. We figured our house contained about $30,000 in clothing. Today that estimate is about $13,000.
We released $17,000 of cash in roughly one year.
Today, in your source control management (SCM) system, you have weeks or months of features, bug fixes and innovation building up. You have paid the salary of the developers, testers, project managers et al who contributed to the effort. Because you are waiting for some date long in the future to release those features, how much cash is in your closet … er … SCM?
ClosetOps – Tools and Costs
Too often DevOps is boiled down to tools. For laundry, our use of a low water low energy washing machine and gas dryer means that even though we run them every day, there is no significant impact on our energy and water bills.
Choosing a tool should be done only in the context of the measures being optimized.
Back in the Real World
In my real job as Director of DevOps, customer satisfaction was the bottleneck to sales. We would work for months behind the curtain to produce a beautiful custom web site. Except it wasn’t what the customer wanted. And it was weeks late. And it was way over budget. And making any changes took far too long.
We changed three things:
- We instituted a weekly show-and-tell meeting and delivered the site to the project manager for their approval in the production environment.
- We automated the developer environment and matched it exactly to the production environment.
- We ensured from the first day of the project that any committed work by the developer was available to be seen by the project manager
We achieved wild success:
- Project manager wanted the account manager involved. Account managers brought customers with them. The value chain was fully represented.
- Customers were thrilled to be part of the process
- Site launches were a non-event. We had been launching daily or more for months while under construction.
- Making updates and changes to the site became trivial
Customer satisfaction measurably improved:
- Account managers more new business with the confidence we could deliver.
- Account managers won repeat business from existing customers who appreciated our transparency and delivery.
SuckOps – DevOps Sucks
My wife has a number of allergies. If I vacuum the house each day, her quality of life increase dramatically. Since wife maintenance is cheaper than wife repair, I was doing this duty and I was tired of the 30 minutes a day it took. In reality, vacuuming was happening every 3 to 4 days.
SuckOps – Automating to the Right Degree
Santa Claus brought us a Roomba last Christmas. It is possible to set this particular model up so that it automatically vacuums sections of the house each day.
I have never bothered to fully automate this process.
It turns out that manually setting the towers is easy. These fences are so easy to move that it was not worth the effort to fully automate vacuuming. So each day I close a couple of doors and turn it on. I then set up towers and move it to the living room and leave for work.
When our kids get home, they reset the towers and run it in their bedrooms, hallway and kitchen. I set it up to cover the office and dining room at night.
SuckOps – Tools Enable Collaboration
Notice that the job of vacuuming is now shared with my kids. My wife can run the vacuum too since she does not have to be in the house with the running vacuum. The tool has enabled us to share this chore.
Choosing tools that foster collaboration between traditional silos is a key factor in tearing down the wall of confusion.
SuckOps – Quality Through Repetition, Not Control
The Roomba does not do the same high quality job as our Dyson. However, because it does it far more often, the overall effect is that our house is more allergen free.
Again, adhering to the key metric: improving the livability of the house for my wife, we can clearly see Santa brought us DevOps with this automation tool.
Santa gets DevOps.
Back in the Real World
The automated development environments allowed the Ops team to share the configuration with developers in a way developers were used to seeing. This is often referred to as Infrastructure as Code https://devops.com/2014/05/05/meet-infrastructure-code/.
By the third week of using this tool (Vagrant + Chef), some developers were making requests in the code to change the configuration. This fostered a closer working relationship between the developers and operators.
Because development of the site was closely coupled to delivery of the site, quality of the deployment process was achieved through simple repetition.
The key pay off, however, was the ability of any developer to work on any customers project with a single command. This one outcome saved a measured 8 hours per week for each engineer!
PaintOps – My Family Understands Me
Does my family understand DevOps? Yup.
Traditionally to paint the interior of our home we would take a couple of days off, buy tons of supplies, move furniture, stress out, eat out … It underscores the problems of large batches again.
My wife suggested that instead of trying to get it done over 4th of July holiday as we had originally discussed that we instead just painted a bit each weekend when time allowed.
Measuring Time
The project took roughly 30% longer total time than it would have. It spanned about 2.5 months. However, time was not the key measure of success. Stress was. Once we go through the first several batches and understood set up and tear down (drop clothes, moving furniture, washing brushes) I would often come home in the evening to find my wife painting a wall.
My wife was able to deliver paint any time she felt like it.
Software engineers working under a continuous delivery paradigm can deliver features whenever the feature is ready.
PaintOps – Surprisingly Lower Costs
The cost of the project was about 10% smaller than the traditional way. This was due to our ability to easily estimate the amount of paint to buy for a small batches very accurately.
Market fluctuation in the price of paint (sales) kept this number from being closer to 15%. There was a certain discipline in not buying a very large quantity of paint when the price was temporarily lower.
PaintOps – A Small Bit of Chaos
Having our living room half old and half new for a couple of weeks is something we could tolerate. Its part of our family culture.
Any paradigm shift in how work is done must match the culture of the organization.
Culture must evolve with, or even before, new models for work can be attempted.
Back in the Real World
A DevOps journey is always a work in progress. We constantly change the way we use tools, the tool, the process etc. This means we occasionally miss a deadline, delete a production server or a root file system.
As part of a cultural shift, the business understood that we were in a far better place than 11 months ago. This shift was enabled by many small wins that built trust over time. Our culture had to be that each mistake came with an effort to ensure it never happened again.
A bit of chaos is necessary to constantly improve. Measuring the impact of each change justifies the cost of mistakes.
HomeOps – Food for Thought
When staring at a DevOps transformation, many executives and leaders are frustrated by the lack of a manual. There is no prescriptive way to do DevOps like there is for ITIL or ISO compliance.
The reason for this is that the only prescription of DevOps is to continuously improve the system by which you deliver value to your customer.
Find something small in your life that you want to improve. Measure it based on the metrics that define “improve.” Change behaviors or introduce a tool and note the difference in your metrics. React accordingly. Repeat.
Choosing small problems leads to solving large ones over time. Soon enough you will feel ready to tackle a problem in your professional life.
You are now free to DevOps.
(See Boyd present on HomeOps at DevOps Connect: CD Summit the afternoon of Oct 8 at Austin Convention Center. Tickets are sold out but walk ups are welcome and there is room)