Last time, Dear Reader, we discovered how the Reverb.com e-commerce platform, which connects musicians who want to buy and sell gear sought to automate its workflow software development process, speeding deployment and bug fixes while satisfying customers who want to trade in an online environment that is as efficient and error-free as possible. Now, the conclusion of how Reverb.com adopted DevOps to achieve its goals.
A Round Trip Through The Reverb.com CD Process
Today, Reverb.com focuses the DevOps mindset on its development process using Chef [https://www.chef.io/chef/] for configuration management and Jenkins [https://www.cloudbees.com/jenkins/about] for continuous delivery and infrastructure automation tasks. Developers use Capistrano [http://capistranorb.com/] for rolling out new code changes and running database migrations, CircleCI [https://circleci.com/] for unit testing code, and DataDog [https://www.datadoghq.com/] for metrics and monitoring. “We also use Logstash [http://logstash.net/], ElasticSearch [http://www.elasticsearch.com/about/], and Kibana in unison for log aggregation and visualization,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com.
In a typical deployment for Reverb.com, an engineer clicks the build button in Jenkins, which triggers a Chef converge on all the web heads. This rolls out any configuration changes that the developers have scheduled to go out, such as NGINX configuration updates or app config updates.
Once this is finished, the system signals the application to perform a Zero Downtime Deploy in which the application reloads without dropping any requests. “If this process finishes successfully, Jenkins kicks off a code deploy using Capistrano and simultaneously deploys new code to our web application servers. If a database migration is scheduled to be run, that will also happen on one of our worker boxes,” says Enger.
This may sound like a standard deployment so far. But that’s where DataDog and the ELK stack come into play. “That’s where we really gain the benefits of CD/CI and DevOps,” says Enger. Every time the development team converges Chef, they use a Chef Handler to submit an event to DataDog. And once Capistrano starts and finishes, another event goes to DataDog. The team has configured Logstash to watch the application logs on the web heads and to send an event to DataDog once the system renders an error. “The single point of information now becomes DataDog because we can see into our config and code changes and immediately know when a bad change goes out the door,” says Enger.
The last thing the development team looks at is the performance logs in Kibana, to see how the application is performing based on the logs it is generating. “With the ELK stack in place, the team can answer questions like: Are there any 500 errors? If so, how many and how often are they occurring? Did development increase or decrease application response times? Do people like the new feature rolled out at /my/feature?,” says Enger.
The biggest hurdle in moving to DevOps is the mental shift you have to make from that of a traditional systems administrator or developer to a more DevOps-based mindset. This means the operations side learns coding and development learns about load balancers.
“If we propagate continuous change in our computer systems, we should also expect to be continuously changing our mental models to adapt to new ways of doing things,” says Enger.
Benefits You Can See Through
Transparency is the number one thing the Reverb.com developers gained from this new CD approach. “Capistrano tells HipChat [https://www.hipchat.com/], “Hey, code is going out!” and I know to pull up my dashboards in Kibana and DataDog to watch for any anomalies,” says Enger. Humans can’t always be expected to communicate to each other efficiently. Computers help the Reverb.com team communicate across the organization through programmed notifications as to when changes go out.
Code deploys are now predictable and automated. No more worrying about a failed deploy due to a bad connection. “When failure does occur, we are able to open up the Jenkins Build output and find the error pretty quickly,” says Enger.
A Nice Surprise
One really positive result that surprised the development team after implementing CD/DevOps is that they learned a lot about their user behavior and traffic patterns. “Now we know that 4.30am is the slowest period on the site,” says Enger.
The team knows exactly when is the best time to roll out big configuration or code changes. They know immediately how deployments affect users and especially whether they ever break critical paths in the site, such as the checkout or signup functions. And the automated approach that DevOps provides enables them to fix bad changes within minutes of their original deployment.
The Affirming Roar of Fan Feedback
Reverb.com recently introduced more automation and monitoring to prevent site downtime and abuse by bots. The feedback: Customers don’t care about DevOps; they care about sites being operational and efficient. But it’s DevOps that makes that happen!