My blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps. One of the seven pillars is continuous integration (CI). The blog explained that the pillars are not silos, but rather foundations that need to be kept in balance. A well-balanced CI implementation performs integrations that are fast enough to fit within the time budget for an end-to-end DevOps pipeline.
Compared to older waterfall processes in which large software changes were integrated occasionally, the DevOps CI concept of frequent, small, incremental integrations has many merits. Quick feedback on frequent integration problems enables a release consisting of many individual changes to be built incrementally with confidence. The root cause of integration problems can be isolated much faster when the changes are integrated incrementally. Since faster integrations are better, just how continuous can integration be?
Contrary to the name, integration is never truly continuous. Even if a continuous stream of changes to integrate existed, typical CI process steps (listed below) require a non-zero amount of time. These steps absolutely interrupt the continuous integration cycle to some extent.
• Merge changes in the integration repository
• Check integration repository entry criterion
• Run static analysis scans of the integrated source code
• Orchestrate the build resources
• Create the build
• Save build the artifacts to artifact repository
• Run unit tests
• Save unit test results to artifact repository
• Release the build resources back to the pool
• Orchestrate the functional build test resources
• Run functional tests
• Perform analysis and assessments
• Save functional test results to the artifact repository
• Remediate any problems discovered during the above steps
• Promote the build to the delivery stage
• Release functional build test resources back to the pool.
For CI to perform as continuously as possible, there are three categories of CI best practices:
- Modularize the product design to support modular CI processing
- Modular code designs, using 12-factor-app principles, allow incremental changes to be integrated.
- Microservice architectures enable each portion of a product to be integrated as available rather than requiring the entire product to be integrated at once.
- Using containers can help to optimize infrastructure orchestration.
- Re-factoring build schedules to remove any unnecessary build dependencies between modules or microservices eliminates time delays.
- All changes are checked into a common trunk branch. This requires the ability to toggle feature changes on and off quickly and easily to enable a release whenever it is needed.
- Optimize the CI Infrastructure
- Vertical and horizontal scaling of both the CI build resources and test resources creates an infrastructure optimized for each CI stage run.
- Predictive orchestration of CI infrastructures enables CI and test resources and processes to be set up in advance.
- Accelerate CI processes
- Pretest and pre-check integration deliverables prior to integration to trunk, to minimize costly remediation delays.
- Pipeline CI build and test processes to eliminate setup delays between processes.
- Use Fail fast to prioritize the most important tests as early as possible.
- Use accelerated testing techniques such as risk-based specific test selection to speed up CI testing.
- Use in-process analytics that can detect threshold exceptions prior to the end of the CI build or test process.
- Remediate problems fast using automated rules for roll-backs or targeted problem reporting.
- Establish a policy for deciding when and how to revert changes that break CI processes so that the pipeline doesn’t get stuck when check-in breaks the trunk.
The above CI practices commonly reduce the time for CI from many hours to minutes per build. While minutes is still not truly continuous, for most practical purposes CI processing in minutes is fast enough to suit even the most aggressive end-to-end DevOps delivery pipeline goals.
What do you think? Do you agree with the above CI best practices? Are there others you recommend?
While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to put in place governance policies that encourage the selection of tool-agnostic IT partners with solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all their IT systems.