We have done continuous deployment (CD) for four years, and enable many of our customers to do continuous deployment with quality assurance (QA). We’ve helped tons of companies adopt and optimize their CD setups. We’ve found that continuous deployment is all about trust — augmented with processes and infrastructure.
To that point, here are some common tips for adopting continuous integration (CI)/CD that I find myself sharing frequently:
Use a CI Tool
This sounds obvious, but you’d be surprised at how many companies try and roll their own. Don’t. There’s no upside for your company—and infinite downside. We love Circle, but there are plenty of other great options. Most are at least somewhat prescriptive when it comes to workflow, which is important when you’re getting started.
Do Unit Testing
It’s always worth reminding everyone that continuous deployment is simply continuous integration applied to deployments. To do continuous integration, you need good unit test coverage. If you haven’t got a solid base of unit testing and continuous integration already, you’re not ready for continuous deployment.
Have Solid Production Monitoring
Bugs and regressions will get shipped; it’s inevitable. Put a system in place, by looking at the data in production, to tell you when a regression or bug was shipped. Tie this into automated rollbacks, so that if a significant performance or key performance indicator (KPI) regression (i.e., signups per minute) appears, your CD system rolls back to the most recent stable version while your team investigates.
Trust Your Team
We talked about the infrastructure requirements, but if you don’t have trust, then continuous deployment isn’t going to work for you. Trusting your team is about giving your developers the freedom to make mistakes, deploy as they see fit and check each others’ code. It is possible to combine continuous deployment with a culture of hierarchical permissions, but we have yet to see that truly work.
Remove Red Tape from the Code Review Process
Your team should review code changes, typically through a pull request, which goes hand in hand with trusting your team. Developers should be trusted to choose the most qualified and insightful person to review their code. One of the most common mistakes among teams trying to adopt continuous deployment is that they only let one or two “senior” developers review the rest of the team’s code, “bottlenecking” all releases. Don’t make this mistake.
Involve Developers Closely in Production Operations
The most common mistake we notice among companies that don’t make a successful transition to continuous deployment is that their development organizations are siloed. Dev and Ops should be joined at the hip. Have your developers deeply involved in the setup and definition of your CD infrastructure, and half the battle is already won.
Shift Left: Test Early and Often
Your team needs constant feedback. Think of your QA objective as getting the right feedback in the right hands at the right time. When it comes to deployments, this means knowing what’s working and what’s broken as the breaking change is made. The sooner you uncover the bug after it’s created, the easier it is to fix. The best companies doing continuous deployment have comprehensive unit test and integration test coverage that runs before any code hits the staging server.
Treat Staging as Your Holy Place
The final pre-production environment is usually known as ‘staging.’ A common mistake is to have developers ship new code direct to staging, while figuring out if the new code is ready to go live. This is a terrible idea. As soon as more than one developer is trying to ship their changes, your Slack channel will blow up with cussing and recriminations. Isolate the pre-production environments to each project, and enforce the “staging as ready to go live” philosophy. This way, integration testing can happen at a self-defined pace, without anyone else’s release being blocked.
About the Author / Fred Stevens-Smith
Fred Stevens-Smith is Co-founder and CEO of Rainforest QA, a quality assurance-as-a-service startup backed by Storm Ventures, Rincon, Andreessen Horowitz, Morado and Crosslink.
