Has DevOps Caused the Re-emergence of Shadow IT?

Shadow IT is back, and this time, it’s in the DevOps toolstack

“One of our clients conducted an audit and found more than 10,000 Jenkins instances across their IT landscape!” This remark from a tooling vendor that helps enterprises centralize their continuous integration infrastructure really took me by surprise. For starters, I was impressed how the company had unleashed creativity at this scale—if the number of Jenkins instances in an organization is an indicator of DevOps. Clearly, its employees have taken the saying, “Don’t ask for permission, ask for forgiveness,” very seriously.

However, there are certain risks with this approach, especially security risks if teams haven’t considered the long-term implication of their decisions. For instance, how do you handle another major security flaw such as the OpenSSL Heartbleed bug? You may have to send hunting squads to track down all the instances of Jenkins (in this case) across your IT estate—including the Linux server hidden away under the desk of the engineer—and patch them. If you are lucky, you can audit all the virtual machines on your chosen cloud where all your tooling instances reside. And if you are really lucky, the entire DevOps tooling infrastructure would have been set up in a DevOps way, using infrastructure-as-code principles with automated patching capability.

Consider the nightmare scenario for IT compliance, security and governance teams: The spectre of shadow IT is amplified due to poor adoption of DevOps! Shadow IT in the application stack has always been a big headache for CIOs; the use of IT systems within an organization without the approval, or even the knowledge, of corporate IT has never gone away. Shadow IT has been historically associated with business users who made the purchase, but did not have the expertise to maintain and upgrade the software to make sure there were no risks to the wider organization. Much has been written about the problem in the application stack and how DevOps ways can help resolve the issue.

However, with the proliferation of DevOps tools in a localized way, the shadow IT problem is now manifesting itself in the tooling architecture of enterprises. With the increasing adoption of DevOps practices and tools in the enterprise, teams have started installing various DevOps tools in a localized and autonomous way. This is leading to the shadow IT problem in the tooling landscape of enterprises. It’s slightly different than the problem that originated in the application stack, but with similar risks and challenges. Access to cloud in the enterprise has made it easier for teams to set up their own localized DevOps tool sets. A sizable program of work in its full DevOps glory can easily use dozens of tools and need to consider aspects of security such as:

1. What is the patching policy ?
2. Are your DevOps tools critical to the business due to the dependency on them for the build and deployment of critical  services that the business requires?
3. How resilient is the tooling infrastructure ?
4. Is there enough knowledge in the enterprise to handle this change?

Essentially, there are four risks of shadow IT due to DevOps tooling, based on a pertinent classification by Computer Weekly:

1. Software asset management: The risk of discovering unapproved and unlicensed software is huge and the CIO could be potentially jailed for sanctioning usage of unlicensed software.
2. Compliance with industry and government standards to demonstrate quality to customers.
3. Lack of testing and change control: Managing cycle of change and releases becomes complicated when there are dependencies on third-party SaaS.
4. Configuration Management: IT teams rely on the veracity of information in the configuration management database CMDB and shadow IT compromises it.

Organizations typically respond to the problem of shadow IT with rationalization, migration or even sunset of the targeted applications—most of which are reactive measures. In the context of DevOps tools, an extreme way to deal with the problem is by taking control of all tools with an “iron fist” and shutting down everything—at the risk of killing off innovation in the enterprise. In the short run, teams do achieve higher levels of productivity by deploying tools in the way that suits their business context and experience of the engineers in the team. However, over time I find that the teams are overburdened by the maintenance overhead of DevOps tooling and maintaining a tooling stack in addition to their software application stack. At times, if they are early adopters in their organization, they tend to be victims of their own success: by supporting other teams with their tooling solutions, almost becoming a tooling platform provider internally, and diverting their attention away from their core purpose of delivering value to their business partners.

One pragmatic way to resolve this is to add an element of centralization or “platformization” of the tooling infrastructure.

Autonomous teams understandably fiercely resist any attempt to centralize services in the enterprise. A few useful ways to gain a balance between innovation and control include:

1. Provide a responsive tooling platform: “You can have any color as long as it’s black” approach to DevOps tooling will only lead teams to finding their own unique solutions.
2. Anticipate tooling needs and build them: Think ahead, track the trends and procure the tools that you expect teams to achieve. For example, if a few teams are already playing with docker and containers, then think a step ahead and look at Kubernetes to manage the containers.
3. Focus on user experience: Think about the ease of subscribing to tools, installation, billing … the entire life cycle
4. Offer help to transition to the tooling platform.
5. Run an internal communications campaign on the tooling capabilities that are available to teams.

Ultimately, a well-thought-out DevOps approach and implementation can actually help resolve the challenge of shadow IT that organizations face, instead of accentuating the problem.

— Aditya Vadaganadam