Engineering Applications for DevOps (Part 5)

This blog series explains how to engineer applications for DevOps.

Topics covered in this blog series:

• Factors to decide whether an application is a good candidate for DevOps.
• Practices to engineer designs for DevOps.
• DevOps applied to enterprise apps and software services.
• DevOps applied to COTS systems.
• DevOps applied to manufactured (embedded) systems software.
• Five levels of application maturity.

This blog covers DevOps applied to manufactured (embedded) systems software. You can read part one here, part two here, part three here and part four here.

DevOps Applied to Manufactured (Embedded) Systems Software

In the late 1970s (yes, forty years ago!) as a young engineer, I was given an assignment to figure out how to accelerate the software development and delivery cycles for an innovative packet switch called SL-10, (predecessor to the more recent model called Data Packet Node [DPN]) manufactured by Northern Telecom. Each SL-10 “node” consisted of millions of lines of original embedded software source code and custom hardware configured with many interconnected processors and protocols interfacing with other nodes, host devices, and terminal devices.

Given the complexity of the system and lack of automated build, test and deployment tools, it was taking more than twelve months to generate each release of the system, and there typically were lots of bugs even after the release. Without knowing much about DevOps (the word “DevOps” was not invented for another twenty-nine years), we proceeded to implement a continuous delivery system that included automated build, test and delivery tools and workflows.

The result was remarkable. Lead times for releases to manufacturing were reduced more than tenfold, from twelve months to one month, and quality, measured as defects found by customers per release, also reduced tenfold. This dramatic shift in performance and capabilities made it possible for the engineering and manufacturing teams to accelerate the rate of innovation and trample the competition. The SL-10 product, and its rebranded DPN version, quickly dominated the connection-oriented packet switch market, and it became the pride of the Nortel corporation and the Canadian nation. Even cooler, I got a nifty president’s pin and promotion which paved the way for my career that led to writing a book and this series of blogs! If this real-life example is not sufficient to demonstrate that DevOps can be applied to manufactured embedded software products, then I don’t know what is.

There have been many other examples published that show how DevOps practices applied to embedded manufactured systems have reaped similar benefits. A notable example was published in 2013 by Gary Gruver in A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware.

That is not to say that DevOps applied to manufactured software embedded systems is the same as applying DevOps to enterprise applications or COTS systems. A paper published on the CloudBees blog, Five Aspects that Make Continuous Delivery for Embedded Different, explained some of the key DevOps practices used with embedded systems.

DevOps Benefits for Embedded Systems

Benefits of DevOps for embedded systems are summarized as follows:

• Retrofitting the huge code base typically found in legacy embedded software systems into a DevOps model is not easy. It is likely to be expensive, but it is almost guaranteed to be a worthwhile effort, especially if you have a longer-term vision and intend for your product to stay competitive in the future.
• Developers of embedded software need access to massive amounts of computer resources for testing their real-time embedded code on powerful hardware simulators.
• Developers and testers need access to physical product hardware to run their binaries for system tests that cannot be run on simulators alone. Lab-as-a-service (LaaS) systems that orchestrate testing topologies of real devices can improve sharing of critical resources. LaaS systems provide a test framework that can be integrated through APIs into a continuous delivery pipeline.
• Long build and test processes may need to be reengineered to reduce serialized lead times into quicker build times made possible by scaling build and test resources.
• Long compliance and conformance test processes for embedded systems need to be accelerated with automation and resource scaling.
• Focus on continuous delivery rather than continuous deployments in which shippable product versions are frequently delivered to manufacturing for verification, but the frequency of deployments to end customers is determined by other business factors.

What This Means

This blog is one of a series explaining how to engineer applications to be most suitable for DevOps. This blog series explains how to engineer applications for DevOps including:

This blog covers DevOps applied to manufactured (embedded) systems software. You can read part one here, part two here, part three here and part four here.

For more information regarding how to engineer applications for DevOps refer to my book Engineering DevOps.