It’s a very exciting time to work in Information Technology – as John Willis* put it:

“DevOps finally proves how IT can be a strategic advantage that allows a business to beat the pants off the competition. This is the moment we’ve all been waiting for.”

I work with organizations that are palpably excited that the software and applications they are creating deliver groundbreaking, mission-critical capabilities to their end users and plan to outsource all their ‘generic’ IT – the services that don’t offer anything unique to their customers’ experience. The challenge though, is that these organizations have a lot of habits and history to deal with; while the business puts pressure on IT to develop and deliver more innovation faster, the increasing criticality of these applications and the evolved complexity and fragility of the systems they run on results in lots of risk and lots of pain when things don’t go according to plan. Here’s how we see this often playing out in today’s enterprise IT marketplace.

Herewith, 6 of the most common causes of conflict DevOps sets out to solve that we hear when we’re talking to our customers:

1. I need a server… now

This is without a doubt the most common complaint we hear: that operations teams, or those responsible for making infrastructure available to development are blockers, too slow and don’t understand the needs of the business. What we see actually happening is demands for immediate drops of physical and virtual servers from teams who not only have literally just heard of this requirement coming down the line but have a pretty full on schedule of planned work too. Oh, and more than likely unexpected system failures and defects are generating a fair amount of unplanned work too having a direct knock on effect to their ability to respond to these demands and their planned work.

Recommendations:

• Culturally something has to happen to ensure everyone knows that they are on the same team and it feels like it. Try involving operations in the development cycles earlier so they have visibility of what’s coming down the line and are bought into the overall goals of the project
• Make it easier to deliver new infrastructure by automating it, creating or using cloud and virtualization infrastructure
• Aim for an on demand deployment capability
• Analyze your teams’ planned/unplanned work proportions and try and figure out ways to eliminate the problems that distract from the goals trying to be achieved – this might be better testing or better monitoring of outages

2. I want access to the production systems

This one usually comes about after the developers have reached a tipping point in their frustration in not getting their needs/demands met in what they would consider a timely fashion. Sometimes it indicates that an organisation doesn’t appreciate or hasn’t implemented workable controls in the operations teams. DevOps doesn’t mean throwing all process out of the window and putting live environments at risk – it’s by no means mutually exclusive with methodologies like ITIL but it is about individuals collaborating better to achieve a shared desired outcome.

Recommendations:

• Read this article: “DevOps Does Not Equal Developers Managing Production”(link)
• Involve development in the definition of your change control processes and set up of any systems
• Set some Service Level Agreements as a team

3. Something is wrong…whose fault is it?

One thing DevOps is very clear it’s not about is blame. The “S” in CAMS* is for “Sharing” – that’s sharing ideas, goals, workloads and responsibility – it’s about being innovative, taking calculated risks, protecting against failure and BEING NICE. That means no finger-pointing and no swearing at each other.

Recommendations:

• Watch this video from AppDynamics: “The Real DevOps of Silicon Valley”
• Read more about running a good, healthy retrospective – and practice it. Here’s an example from the UK Government Service Design Manual
• Consider implementing tooling that will enable you to pre-empt failure or fail smartly. And that will help you identify the root cause fast

4. So many alerts they are meaningless

We spend a good proportion of our time working with companies to implement systems for smart failure or that can pre-empt failure as mentioned above. This is one third of our devops utopia triumvirate. The kind of systems that can deliver this capability are the new breed of Application Performance Management solutions like appdynamics. One of the challenges we often find though is that enterprises already have monitoring systems. Lots of them. Monitoring lots of different things. All producing a multitude of alerts. This is when we see alert fatigue. This is not something I believe is, or have indeed witnessed to be, relevant only to the healthcare industry. When someone receives a lot of alerts, of which a very large proportion are meaningless, it becomes highly likely that the critical ones are missed.

Recommendations:

• Review and consolidate your existing systems
• Take some time to really understand what alerts matter and adjust your policies according – and consider this an ongoing task to tweak your systems as your situations change
• Some DevOps proponents recommend alerting development on system outage immediately – I think this could work, if development are bought into the process and not thinking they are simply being blamed (see 4 above) – sharing the load is definitely a good idea, as long as the process makes overall responsibility clear and forces resolution

5. I don’t have time to save time

We hear this one a lot, particularly when we start to talk about business cases for automation tools. The two things driving DevOps (1: the increased rate of innovation and 2: the fragility of today’s evolution of IT infrastructure) together result in very high workloads and a LOT of unplanned work. Throw a recession and budgets that have been squeezed hard year on year on year and what you get is a lot of late nights and a lot of missed targets. DevOps is designed to streamline the throughput of work at a cultural, interaction and application based level – diffusing the pressure building up in many IT departments today.

Recommendations:

• Consider ringfencing some budget so invite in an external consultancy to perform an audit or assessment of your current DevOps state – you may feel they won’t tell you anything you don’t already know, but you will have someone documenting your position for you and giving you justification to take the next step in your transformation
•  Make time – find a senior decision maker who will listen and mandate a project where you can prove something – for example the savings made by automating a piece of a process

6. Our hero is a bottleneck…

This one is particularly common when we are talking about release and deployment automation. And if you’ve read “The Phoenix Project” * you’ll recognize Brent as being one of these. What tends to happen is this: one day an engineer decides that building servers is a tedious, manual process and that they could write a quick script to make that bit a bit quicker. The next day, they write another one. And another one. Until, over a period of months or even years a whole bunch of scripts have been written automating lots of parts of the process – I like to call this a ‘script monster’. The problem is that even though the script monster works pretty well most of the time, when it fails it’s virtually impossible for anyone other than its creator to troubleshoot. So when there’s an outage at 3am, you have to call ‘Brent/the creator/Frankenstein’ and he breezes in and saves the day. But what if he isn’t there one day?

Recommendations:

• Create an environment embracing autonomy, mastery and purpose – where ideas are heard and acted upon. Often these ‘script monsters’ wouldn’t have been born if Frankenstein had felt he had alternatives to explore
• Understand motivations: yours and your colleagues’. Some people are particularly motivated by certainty at work and if they have made themselves indispensable by creating something like a script monster, they may feel threatened if someone proposes to take it away from them
• These heroes are often genuinely some of the most talented members of the team, so if they are resistant to change, talk to them about exciting new projects they can be involved with – make them the master of a shiny new system that will multiply the frequency of releases ten – one-hundred fold by allowing others to push them through an automated process – deployment on demand

In Summary

DevOps isn’t voodoo, a silver bullet or automatic. But it is a drive to encourage a different, positive and innovative way of thinking to address a common set of issues affecting the experience of living in, working with and using IT in the world we live in today. Be DevOpstastic.

What other causes of conflict have you experienced that DevOps can resolve?