The topic of DevOps has been a topic for many a debate over recent
months years. Without adding fuel to the fire I’d like to present a slightly different take on the DevOps culture.
The phrase ‘what if I was hit by a bus?’ can often be heard within the vicinity of software development teams. It relates to a measurement known as the ‘bus factor‘. Individual talent should not be discouraged, however when it becomes silo’d that talent translates to risk for a business. Try to think back over your career – do the following sound familiar?
“I don’t know how to do that, X usually does that”
“X is away at the moment so we’ll need to put that on hold”
“X left yesterday and didn’t cover this in their handover”
If you have read ‘Phoenix Project‘ – think of Brent…
Using the fated bus driver I’d like to take us on a journey (excuse the pun) through a DevOps culture.
In-sourcing the bus driver
You work for a small green minded startup. Being green minded and wanting to help your staff, you decide to buy a bus to get everyone to work in the morning. Realising that you don’t actually have a dedicated bus driver, you decide to take it in turns to drive the bus in the morning. Everyone gets involved, everyone knows how to drive the bus, everyone knows which route is best.
Don’t over engineer your processes. At this stage you’re already exhibiting DevOps characteristics!
The talented bus driver
Your small team has performed really well, people like what you do and now your team isn’t quite so small. There are now over 50 people on your bus in the morning. It wasn’t long until you realised that Sarah is the best bus driver you have. Inevitably, Sarah usually just takes up driving the bus each and every morning.
If Sarah is off, Mike takes over but he often makes people late for work and tends to hit a lot of parked cars.
Furthermore the bus recently broke down when Sarah was off, Mike and your bus passengers all took a look at the engine and tried to help but there seemed to be a ‘magic’ piece of string holding two parts together.
When they consulted the bus manufacturer manual, this piece of string was no where to be seen. Mike had to call Sarah whilst on holiday!
Mike calls the bus company to ask for a replacement bus, to be told it will be 2 hours until a second bus arrives. The team sit and wait patiently.
Try to identify your tipping point. Some key indicators are reliance on individuals, information silo’s, phrases such as ‘oh I didn’t know it functioned like that in production’, production environments start to diverge (without communication) from development environments.
How many production environments have the web server, app server, database, load balancer et al all running on the same machine? Is this how your development teams work? Does this differ to the view of the operations team?
Your bus stops
We’ve identified a few problems but lets focus on the positives. Firstly, let’s celebrate actually spotting the problem, it is very easy to continue with the status quo. Our second success was the reaction of our bus passengers. When the bus broke down, instead of complaining, the bus passengers tried to help mend the bus.
When and where your bus stops is less important than who it stops for. It might be stating the obvious, however if you wish to move towards a DevOps culture you need talented team members. You need people that care for the organisation and their profession. A desire to share knowledge for a greater goal.
If your team are not interested or do not care, then no matter how much you say DevOps you need to boost their interest first.
Hopefully you’ll start to hear your teams utter phrases such as “What does production look like?”, “You could do this slightly differently you know”…
What if your talented bus driver has an RTA?
There are no doubting Sarah’s talents – I mean come on, she was able to fix a bus engine with a piece of string, knew the best bus routes and always got your team to work on time…
What would we do if Sarah left or was, somewhat ironically, hit by a bus?
Could your organisation raise the value of sharing and encourage knowledge share?
Assuming you’ve got the right people on the bus, you might also find that other staff members have ideas for improvement when you explain the current process. Your Sarah(s) should not fear sharing their knowledge and should be rewarded/encouraged for doing so.
One of your passengers may even know a faster journey to work or a better way to fix the bus engine.
The bus seating plan
Although you didn’t enforce it, you start to find groups forming on your bus journey, all your teenagers are sat together at the back playing music on their phones…..
Don’t let job titles restrict conversations. Water cooler chats are incredibly productive and help teams share ideas.
Replacement bus service
When the bus broke down Mike was able to order a replacement but it had a 2 hour turnaround time.
Would it be better if we always had 2 or 3 buses running? If each bus had enough capacity for all 50 passengers we’d probably always make sure people were on time for work.
It would be even better if all bus drivers knew the routes that Sarah used, all buses had the same piece of magic string and all bus drivers knew what the magic string did.
Something my team often hear me say is ‘Design for failure, don’t just react to it‘. When designing your applications and services try to think how the system can accommodate failing services better rather than what actions you might take in the event of failure.
The same might be said for your human resources. Have your teams got a shared knowledge? What would your bus factor score be?
Try to keep your passengers informed of your progress – you could let them know their ETA for work. Informed passengers tend to be more engaged and understanding. For example when telephoning a business, do you prefer to know what position you are in ‘the queue’ or would you rather sit on the line waiting without knowledge of when you may be spoken to?
Monitoring might be considered a key technical section of DevOps adoption. There are many, many tools that can help with this such as StatsD, Graphite, Librato, New relic, AppDynamics etc but whichever medium you choose, try to make sure your ops stats are in full view of your various teams not simply your ops staff. Get them in front of your dev team, support team, sales team, HR team they impact every member of staff.
Hopefully, you will be surprised at an increased level of employee engagement. When ops are in war rooms fire fighting problems, each team member will understand the pressures they are under, some team members will also express a desire to help.
You have arrived at your destination
At this stage I may have flogged the bus concept a little too far. Are the concepts here a guide for the adoption of a DevOps culture? Does the ‘D’ word have to be attached to the guidelines above? I’ll let the community decide.