Preparing to adopt DevOps can be a challenge for organizations of any size. Scaling DevOps and bringing it to an enterprise demands a special strategy. Use these tips to build yours.
DevOps principles apply to all organizations. Everybody has business goals that can be met and exceeded by following DevOps practices. However, an organization’s size will inevitably impact the level of difficulty it will face during implementing and scaling DevOps. A large enterprise organization will encounter more daunting obstacles than a smaller organization, even if their problems are similar in nature.
Regardless of your organization’s size, you are probably wondering what to expect during your DevOps transformation and asking yourself which strategies you can use to overcome any complications. Luckily, you now have “The DevOps Handbook” to turn to for advice. This book is an excellent resource that distills the most important insights learned from difficult experiences into straightforward steps and solutions for overcoming common DevOps hurdles.
DevOps organizations move much faster than many other organizations today without sacrificing quality, performance or security. This blog post will cover some helpful strategies from “The DevOps Handbook” that will help you overcome challenges your organization might face during its shift to DevOps.
Changing Your Organization’s Culture
The first hurdle in the DevOps process is managing culture and expectations. Scaling DevOps throughout the organization requires changing how your business views itself. It also involves rethinking your process for delivering value to the customer. The goal is to get the changes you make in your products and services to the customer faster. In technical terms, this boils down to how often deployments happen. The DevOps process uses positive feedback loops to deliver value to the customer quickly.
Related Content:
What is an Agile IT Organization?
Everything Ops: Are DevOps Principles Being Applied Too Broadly?
To achieve this goal, “The DevOps Handbook” says that it is necessary to shift your focus from cost-optimization to speed-optimization. Optimizing speed does not require a massive, top-down reorganization. Rather, it means giving teams more autonomy. Different skills (such as development, operations and information security) should be embedded into existing teams. This will make teams become self-serving, which will enable them to develop, deploy and operate their software more safely and effectively. Majorly successful organizations such as Netflix and Amazon attribute their success to this approach.
However, building self-serving teams is just one part of the culture change. You also must change how your people think about themselves and their work. Let’s take Toyota, the pioneers of the lean manufacturing process, as an example. The “Toyota Way” has propelled the company’s success, as Mike Rother wrote about in “Toyota Kata.” In the book, he explains that while structural reorganization is an important step toward overall improvement, it is also critical to invest in the betterment of your employees. You have to build upon their technical work abilities and develop the habits of continual learning and experimentation. Toyota accomplished this by using what Rother called The Improvement Kata, which is a model for achieving goals that is based on coordinating theory and evidence.
DevOps organizations integrate continual learning and experimentation into their day-to-day work and decision-making. Verifiable experimentation, such as A/B tests and cohort analyses, advance business goals. The challenge is building the data-driven learning framework. Product owners and other decision makers need a variety of business telemetry. Accordingly, engineers must collaborate and implement whatever is necessary to provide it. Everyone can—and should—use this telemetry to understand what is happening with the business and IT at any point in time.
Etsy is a good example of an organization that has implemented telemetry well. According to “The DevOps Handbook,” in 2011, Etsy started tracking 200,000 business-layer metrics, displaying 30 of the most important metrics on its deployment dashboard. By 2014, the company had tracked more than 800,000 of these business metrics. This high level of dedication to telemetry made its engineers’ jobs easier with quick business and technical documentation, especially after deployments.
The next hurdle is combining telemetry with the practice of continuous learning and experimentation. The two practices work best together when the process is moving as quickly as possible.
Faster Iterations
Addressing speed-related challenges requires more of a technical effort than the others. DevOps requires fast feedback loops at all levels, which begins and ends with a new set of engineering practices.
Trunk-Based Development
First, you should start using Trunk-Based Development (TBD). TBD is a model for source-control branching, in which multiple developers work together on code from one, individual branch. With TBD, developers check in their code to trunk one or more times daily. Using this method has powerful knock-on effects. Automated tests run against the entire system, so it is immediately clear if something is broken. TBD also reduces long-running feature branches that may eventually create difficult merge conflicts or introduce unexpected bugs.
Another great thing about TBD is that it naturally keeps changes smaller, which means that they are easier to review, test and reason about, all while keeping your code in a releasable state. It can also help you reduce your total number of environments, which can easily save tens of thousands of dollars every month.
Automated Deployment
Next, you should automate your deployment process. This requires automating as many of the intermediate steps as possible, including developer workstation setup, test environment setup, environment promotion, the deployment itself and telemetry collection. This is a lofty goal, but taking it to the extreme creates powerful business outcomes. More people are able to deploy more often. Deployments become smaller, safer and more routine, and the number of production bugs is reduced.
Making this happen is no small task since everybody involved must adopt the “automation first” mindset. Start by automating the more error-prone and repetitive tasks. The effort will be worth your time, since automation increases throughput and reduces errors.
Etsy is a big fan of automated deployment. According to an Etsy case study from 2014 that can be found in “The DevOps Handbook,” “deployments are performed by anyone who wants to perform a deployment, such as Development, Operations, or Infosec. The deployment process at Etsy has become so safe and routine that new engineers will perform a production deployment on their first day at work — as have Etsy board members and even dogs!” You may not want actual dogs in your control room, but you certainly want that level of confidence and satisfaction.
Integrating Information Security
This amount of automation sounds fantastic for developers and operations, but what about information security? For them, it’s not necessarily true that topics such as compliance and security are treated as afterthoughts. DevOps is successful because it uses quick feedback loops to identify and fix problems early on in the software development process, but information security and compliance usually come into the picture at the very end, creating huge blockages and problems for everyone involved.
Information security is another skill such as QA or operations, so it can be embedded in cross-functional teams. Then, teams can work together to address known security issues, such as testing for package vulnerabilities as part of CI. Testing for proper secrets, such as a database connection string, can be handled with automation, too. Teams can also integrate static analysis to detect possible problems before the code is executed.
Continuous delivery practices also apply here. Security-related metrics such as total password resets, potential SQL injections, XSS attacks, core dumps and other strange events should be integrated into the deployment pipeline. Doing so provides visibility and immediately actionable information to the entire team during incidents.
Bottom-Up Scaling
Large enterprise organizations are structurally complicated. There are many leaders and smaller teams, each with their own mission and culture. As such, trying to change things top-down is a big mistake—it simply does not work. Scaling out from the bottom-up is a much better approach, and there are a few ways to accomplish this.
Starting small is the best advice. We recommend that you find a small group of like-minded individuals and give them technical autonomy, goals and your trust to improve the business. This will likely help them tackle technical debt issues that have hindered them in the past. They’ll try new approaches and learn down in the trenches.
Unfortunately, there is no such thing as DevOps boot camp—only principles and technical practices. It’s up to everyone in the team (and the organization in general) to figure out what DevOps means to them. Hopefully, their results will speak for themselves. Naturally, other teams will become interested in how they work and think. In time, some team members may even be ready to train their peers.
Ross Clanton at Target uses a “technological innovation center,” which is more commonly known as a DevOps Dojo. The Dojo coaches help teams across Target’s IT department elevate their states of practice. The “30-Day Challenge” is their most intensive training. Blocked development teams bring their work with them to work alongside dedicated coaches and engineers. The goal of the challenge is to unblock your team in less than 30 days. According to Clanton, taking part in this challenge has helped many teams finish work that would normally take months in just a matter of days. Completing these challenges has solved many internal problems, and has ultimately made the organization as a whole more productive.
Conclusion
There is more than one way to scale DevOps in your organization. In this blog post, we have reviewed three of the main DevOps principles that will help you along your journey. First, you need to begin changing your organization’s culture and organizational structure to allow for a “divide and conquer” strategy. Next, establish protocols that will help you find errors early. Doing so will increase the speed of feedback loops, which will make for quicker delivery of customer value. Third, invest in continuous learning and experimentation of new ideas.
Using this and other advice from “The DevOps Handbook,” you can work through your DevOps transformation with confidence. When in doubt, we suggest that you consult this excellent resource for more helpful tips.
— Rami Sass
