Intelligent DevOps for New Age Digital Systems

Transforming DevOps by leveraging the power of analytics and autonomics

In a previous blog, I have written about how Quality Assurance is getting transformed to Quality Intelligence using the power of big data analytics and autonomics.

In this blog, we will discuss how DevOps solutions of the future can become more intelligent by taking advantage of these same emerging digital technologies such as autonomics and analytics.

Technologies such as predictive analytics, autonomic computing (autonomics), artificial intelligence and robotic process automation (RPA) are already driving unprecedented levels of automation in many other businesses. Combined, these three technologies are driving the next generation of automation—what is commonly referred to as “intelligent automation.”

Intelligent automation goes beyond basic automation and orchestration of manual tasks to incorporate adaptive self-learning—leading to fully autonomous systems. The figure below (courtesy IRPA) illustrates the evolution of automation as it moves toward full autonomy. The Google car (and other similar solutions) is a great example an intelligent automation system, capable of learning from its environment and adapting behavior accordingly.

Automation in Traditional DevOps

Digital systems are driven by new age technologies such as social, mobile, big data, analytics, cloud, internet of things (IoT), artificial intelligence, augmented and virtual reality, genomics, etc.  These technologies introduce significant change and complexity to how modern systems are built and deployed. Businesses that leverage these technologies to transform themselves into cognitive enterprises demand new and intelligent approaches to the underlying processes and techniques to support them. Such processes and techniques need to be more than agile; they need to adaptive and capable of reacting dynamically to frequently changing circumstances.

Let’s look at some examples of current DevOps techniques and how they will need to evolve to support dynamic digital systems. Automation within traditional DevOps solutions is generally limited to scripting and orchestration as shown above.  Automation adoption levels vary, due to various reasons ranging from complexity to skills and organizational challenges. Maintenance of such scripts is itself sometimes a bottleneck, as applications and environments change rapidly, the scripts need to be updated manually and often become obsolete. New age agile digital businesses, by contrast, demand automation that is able to adapt dynamically and self-heal on demand.

Further, classic DevOps automation techniques are typically driven by static predefined rules. For example, the criteria for promotion of an application build through different stages of the pipeline is often statically defined. This is a limitation for new age systems where the criteria needs to be dynamic and may vary depending on multiple circumstances. The automation system needs to able to look at past data, keep learning from current data and make adaptive intelligent predictions about the right course of action.

DevOps for internet of things (IoT) systems is very different than that for traditional software systems. The target environment for production deployment in consumer IoT is geographically dispersed, typically not configuration controlled, may have unreliable network connectivity and may even be fragile. Further, IoT systems generate huge amounts of data that necessitate robust data mining and self-learning (adaptive) techniques not provided by traditional lifecycle automation tools.

Similarly, customer experience (CX) is a key new metric for digital systems that transcends traditional DevOps metrics such as release velocity and quality. CX data is unstructured, fuzzy, voluminous and volatile. CX-driven DevOps (or CX-Ops) is an emerging discipline that requires big data analytics and cognitive approaches (such as natural language processing or NLP) to decode meaningful insight from such data.

Hence, as digital enterprises evolve and businesses demand greater agility and flexibility, the DevOps function to support such systems will need to transform as well.

Next-Generation Intelligent Automation Landscape

Before we jump into what intelligent DevOps would look like, let’s first look at the different types of automation systems being adopted in the industry in general. The following types of automation are described as part of the “intelligent automation continuum”:

• “Systems that Do”: These are basic automation systems that replicate human keystroke actions and static (pre-defined) rules-based activities. They also take advantage of descriptive analytics that reflect past trends and patterns. Examples of such systems include speech and image recognition.

• “Systems that Think”: These use algorithms and intelligence to find meaning in data, handle judgment-oriented tasks using diagnostic analytics and make recommendations based on patterns. Examples of such systems include natural language processing and recommendation engines (such as email spam filters).

• “Systems that Learn”: These understand context, interpret and dynamically adapt based on circumstances; they typically leverage predictive and prescriptive analytics to solve problems independently. Examples of such systems include self-driving cars and neural networks.

A Peek at Intelligent DevOps

So, what would transformed intelligent DevOps look like? Intelligent DevOps automation would take advantage of cognitive and autonomics systems to enable smarter adaptive lifecycle automation based on analytics. In my previous blog, I discussed how QA can take advantage of big data analytics. Intelligent DevOps, to a large extent, depends on such capability.

Based on the above model, let’s look at the different types of DevOps automation we can create:

DevOps Systems that Do: This includes traditional DevOps automation systems (e.g. for continuous integration and testing, continuous deployment), as well as pipeline lifecycle automation that are based on static rules (e.g. traditional release management automation)

DevOps Systems that Think: This includes advanced automation systems such as:

• Automated automation systems, for example, generation of automated test cases from manual tests (or test models) using NLP, generation of virtual services based on request-response data logs
• Self-healing automation, for example virtual services (or test scripts) that can auto-update based on change in application endpoints (or behavior)
• Monitoring of IoT systems, for example “smart homes,” which require continuous use of diagnostic analytics to mine huge volumes of data to understand failure modes and prescribe recovery techniques
• Automated verification of system requirements based on customer experience analytics
• Automated generation of test scenarios based on analytics on production logs

DevOps Systems that Learn: This includes sophisticated test automation systems such as:

• Adaptive continuous delivery pipeline—Learning systems that analyze past data to manage the pipeline based on dynamic rules. For example, correlate code quality and defect detection and slippage patterns to dynamically decide what tests to be run and what gates to be enforced (vs enforcement of statically defined gates) for different teams and products for promoting application builds
• DevOps process optimization based on insight across the life cycle. For example, correlation of production log data with past code change data to determine the level of failure risk in different application modules. See more on this in the section below.

We embody the evolution to intelligent DevOps in CA’s maturity model (shown below), which also aligns with the RPA maturity model described above. Beyond the continuous maturity level, Intelligent DevOps is the driving force behind continuous optimization and improvement.

Cross-Life Cycle DevOps Intelligence

Intelligent DevOps enables us to perform process optimization based on analytics from data correlated across the system life cycle, from planning through operation. Each process area generates a great amount of data that is typically analyzed within the process (and sometimes organizational) silo.

While such analytics is useful in itself, correlation of data across these process areas (see figure below) may be used to provide a wide variety of intelligent lifecycle insight (and process improvement possibilities), such as:

• Correlating configuration data analytics with code change and defect analytics helps us pro-actively identify failure modes related to code and infrastructure changes.
• CX analytics (from the Operations process) can be used to validate requirements in the plan and define processes
• Production log analytics may be correlated to test log data to identify missing test cases

In the spirit of the “Continuous Everything” philosophy in DevOps, we envision a new solution stream around “Continuous Insight” where analytical insights are generated and acted upon continuously (and autonomously) as processes execute.  We will discuss more on the subject of Continuous Insight in a future blog.

In conclusion, we believe that intelligent approaches described above will be infused into every aspect of DevOps going forward and revolutionize the way DevOps is performed. Stay tuned my friends, and may all our DevOps be intelligent!

— Shamim Ahmed