Engineers at Internap, a hosting company and public cloud vendor designed a new OpenStack-based cloud platform and development environment. DevOps.com tells the story of Internap’s DevOps cultural evolution, which grew virally, interwoven with the company’s development of PrivateStack.
Challenges in Internap’s Common Development Environment
Challenges that slowed Internap’s cloud development process triggered a hunger for change and for a new / altered development scheme. “The Internap public cloud product is essentially composed of micro services that make cloud resources available to our customers,” says Mathieu Mitchell, Senior Software Developer, Internap. Internap engineering teams tested their cloud services simultaneously with each affecting the other and transferring adverse effects to production or pre-production. Internap needed to let the billing team test their services, integrated with a duplicate of the production software and environment, without affecting the virtualization team.
These challenges were rooted in a commonly shared development environment for all the engineering teams and team members, which introduced tedium for engineers as they worked to build and deploy stable code.
When sharing a common development environment, each engineer’s tests depended on environment consistency and reliability. When engineers / developers tested two changes at the same time, there was no way to tell which change introduced a regression. The most reasonable way to address this prior to PrivateStack was to push all changes into the environment and dedicate specific engineers to identify and fix the issues that occurred.
The challenge with dedicating specific engineers to troubleshoot these regressions was that they had a diminished context to work with when compared to the original developer’s understanding. The team that introduced a change needed feedback from the environment and to shoulder responsibility for fixing the issue. This would result in a stable codebase and the ability to write more thorough, automated tests.
“This is why we created PrivateStack–to allow us to independently test a single change, integrated with other services, in an environment that is the equivalent of a private production setup,” says Mitchell. PrivateStack enables development teams to innovate while ensuring that changes behave correctly in production. This in turn enables CD and speeds development.
DevOps Culture Leads Teams to Success
As Internap developed PrivateStack, engineering observed a DevOps belief system spreading across the project and the teams. “At first, it was only a minority among us, mostly people with an Agile background, who advocated the Continuous Delivery approach while we worked on this project. We really believe in delivering value to our customers as quickly as possible. To achieve Continuous Delivery, we needed people to understand that automating everything was the way to go,” says Mitchell.
As more engineers realized that they were the solution, they became increasingly motivated to enhance processes, create the new development environment, and use CD to develop the Internap public cloud product.
Results with PrivateStack
PrivateStack enables Internap teams to share the same code and system configurations while individual developers have each their own private production environment to test their software without affecting others. “We heavily leverage virtualization to be able to recreate our environments, since buying racks and racks of hardware to make this available to all of our developers would be cost prohibitive,” says Mitchell.
With PrivateStack, Internap isolates production issues during development, keeping R&D efficient and slashing pre-production troubleshooting time. “PrivateStack also improves our time-to-market, is much less costly, and most importantly, reduces the chance of any issues reaching the customer,” says Mitchell.
DevOps Culture Wrap Up
To finish the PrivateStack project and foster a DevOps culture at the same time, Internap’s internal core of CD true believers lead by example. Mitchell and his colleagues adhered to consistent principles while hearing out other team members on their concerns. This helped them to encourage the larger engineering department including billing and virtualization teams to adopt a DevOps and CD frame of mind.
“We dedicated two people to drive this initiative, along with a few others who believed in the Continuous Delivery approach but were not involved directly,” says Mitchell. That was enough to get the job done. Now the vast majority of the engineering department is targeting CD.
“We are eager to share our PrivateStack platform with the open source community to enable other developers to run production-like environments for development in their day-to-day operations,” says Mitchell.
