“Mixcloud provides audio hosting and distribution for millions of podcasters and DJs,” says Matthew Clayton, CTO & Co-Founder, Mixcloud. Mixcloud’s developers program the site in python (Django) and javascript, and develop their native mobile apps in Java and Objective-C. The Mixcloud team grew its mammoth Django monolith application over a five year period. Mixcloud’s primary server infrastructure supports this application, which makes up most of its code base.
The Legacy Development Workflow & Goals For The Future
The Mixcloud team used to code its application on a branch, deploy the branch, check for stability, merge it into the master branch and redeploy it. “We used to branch the code on git, then push this to github so that all the team could see the branches developing in parallel,” says Clayton. To check code stability, the Mixcloud team used to simply watch to see whether the site crashed on a single server. “We originally didn’t have any tests, so this was really the only way we could see whether there was a problem,” says Clayton.
The old overall development methodology also produced challenges every time Mixcloud hired new developers. “Adding developers who didn’t understand the entire stack massively increased the likelihood of bugs and the risk of downtime,” says Clayton.
When Mixcloud went looking for a DevOps / continuous delivery approach, it had in mind to increase the quality of the code its developers produced and to increase the team’s confidence in the code they shipped to production while overall having faster development.
New Workflow, Bumps In The Road
Today, Mixcloud hosts and reviews its code using github.com. The company uses Slack for internal communications. The team uses getsentry.com for exception handling and Asana for product feature and bug tracking. “We use Codeship for running Linting (python/JS/Templates), JS unit tests, Python tests, Selenium integration tests, and database integration tests,” says Clayton.
Python/JS/Templates enable code linting by leveraging tools such as pylint, jslint, and Django-extensions to perform static analysis and to lint the appropriate code.
But the path to DevOps enlightenment and improvements is never ending. It took some time for Clayton and his team to understand how all the working components affect each other as they integrate together. “We still have not included database migrations in our release process. This is still a manual process,” says Clayton. Mixcloud will eventually add this too to its list of automated endeavors.
Navigating The DevOps / Continuous Delivery Maze
At the very start of a round trip through the new development process, Mixcloud developers will open up a pull request on github.com and assign it to another team member. This action automatically sets a complete build in motion and deploys a staging infrastructure. “The assigned developer will now review the pull request using github, and also the running code. Once this review process is complete, the branch merges with the master,” says Clayton.
This will next set off a deployment in production. When the team is ready, they will run a single command to activate the new code for each and every user. “The whole test and deploy process now takes about three minutes. This enables a rapid iteration,” says Clayton.
Benefits & Surprises
Two of the biggest benefits for Mixcloud were adding code linting and pull request reviews. “These helped the team become more aware of changes across the entire code base, and eradicated entire classes of bugs for what is essentially a fixed, one-off engineering cost,” says Clayton.
When compared with the legacy development methodology that could introduce new bugs every time a new developer came on board, the Mixcloud team now only introduces bugs of a magnitude that could create website outages and downtime in the rarest of cases.
But the major unpredicted upside was how the automation and the reduction in manual testing added to the productivity and development speed. “Surprisingly, there have been massive improvements to interaction speed among team members. We are seeing productivity and developer happiness that are significantly better than before the change,” says Clayton. The ability for the development process to refactor huge amounts of code with confidence has also made great strides forward.