DevOpsQA NJ held our January Meetup on Pipeline Patterns, with Max Griffiths (@_maxamg), Max Lincoln (@devopsy) and Gary Stafford (@garystafford) from ThoughtWorks  as speakers. The meetup took place at the sensational ThoughtWorks NYC location, with welcoming, open meet-and-greet area and a huge presentation area with bright, distinctive wall paintings.

Following speed networking with pizza and beer, the discussion kicked off with a brief intro from the speakers, who each has been in the DevOps space for 10+ years. They gave a high-level overview of ThoughtWorks and some book recommendations including, “Continuous Delivery” by Jez Humble, “Release It!” by Michael Nygard and “The Phoenix Project” by Gene Kim.

So what is the issue that we’re trying to solve? We are trying to gain an ability to make decisions early on to impact how our pipeline is going to scale, with the focus on culture, feedback and governance.

Max Lincoln gave a good definition of a “good” pipeline. It should have incremental QA, fast feedback, visibility of the progress and a consistent, auditable software delivery process.  It’s a good idea to be very selective with how you name things in the pipeline. “The purpose of the pipeline is to kill the build as soon as possible,” he said. As the pipeline gets longer, quick feedback becomes more difficult.

Next, each speaker addressed how the pipeline can scale in three difference directions: X (more stages between commit and productions); Y (more projects/applications); and Z (concurrent releases). Max Griffiths gave overview of X-Axis patterns which include Linear, Parallel and Layered (with artifact promotion and release channels). Linear is the simplest pattern, which works for simple applications. With a parallel pattern, you start to parallelize where possible; however, that might create dependencies and make it difficult to reorder. The layered pattern encourages parallelizing but prevents direct dependencies between stages.

The layered pattern is further enhanced by adding a separate artifact repository for each layer, which makes artifact retention policies a lot easier. You can use bronze, silver, gold as names for the layers to easily understand how release candidate progresses from one stage to another.

Gary Stafford introduced Y-axis patterns which include joint labeling, aggregate labeling, independent pipelines and VCS labeling. among others.  Gary used the Microservices example to demonstrate how to do continuous delivery with loosely related applications or services.

Max Lincoln spoke about concurrent releases with Z-axis patterns. As per Max,   instead of a branching strategy, dev needs a merging strategy, especially if a release branch gets created six months in advance. “If you have Jenkins do 1,000s of builds a day, it does not mean you have CI,” he said.   

The speakers concluded by each introducing an extra pattern which they referred to as “different dimensions.” The parting word of advice was to keep CI (whether it’s Jenkins or GoCD) in source control and test your pipeline continuously.   At the end we had a great Q&A session and a very warm, open discussion.

Looking forward to our next session.

If you would like to learn more about ThoughtWorks and/or Pipeline Patterns, you can continue the conversation on the GitHub page: https://github.com/thoughtworks/PipelinePatterns