“Reverb.com is an e-commerce platform that connects musicians, enabling them to buy and sell used gear and instruments,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com. As an ecommerce site, it is critical to Reverb.com that its workflows such as checkout, offers to purchase, and messaging work in tandem with its selling workflows, which include creating new listings and managing orders.
The Reverb.com team also develops a free, comprehensive online price guide and a complex in-house content and curation system that allows them to hand pick and recommend the best gear for their customers. In the Reverb.com marketplace, all software components must run smoothly for customers to have the optimum research, buying, and selling experience when conducting trade on musical equipment.
To create those components, Reverb.com developers code primarily in Ruby on Rails on Macs. The etailer runs Linux in its production environment. “Our backend stack includes PostgreSQL for our database, ElasticSearch powering our search engine, and Redis for queueing and key value storage,” says Enger.
Not So Good, Old-Time Deployment
Prior to DevOps approaches, Reverb.com deployed code from individual user workstations. “For configuration changes to our infrastructure we had to use SSH to connect into production. All while our fingers were crossed, of course,” says Enger. This loose-knit deployment process presented Reverb.com with some obvious challenges.
“If a deploy failed due to an engineer being on a bad internet connection, we could potentially have had two updated hosts while three were still running an older version of code,” says Enger. This was especially problematic because these deployments often included database migrations.
There was also no real way to know when a deployment failed or who started the deployment. “The process was devoid of any deployment log records, so errors and anomalies were harder to spot when the deployment was swallowed in the developer’s terminal,” says Enger.
These risks made it possible for Reverb.com to face an even bigger risk: a lapse in faith on the part of its customers. Reverb.com’s reputation as a marketplace hinges on the perception of trust among its customers. Every aspect of the organization generates trust. From how Reverb.com interacts with customers as people, to how its systems present to users, the firm wants to exude confidence. “If we experience outages and slow performance, we may erode our trust with our audience,” says Enger.
Goals For DevOps
Reverb.com had many goals for DevOps in the hope that it would meet their challenges to fast, scalable, reliable delivery of code and configuration changes. Reverb.com needed to know when a change was going out. The firm needed to be able to deploy at a moment’s notice.
“All processes must be behind the click of a button or an SCM post/pre trigger,” says Enger. All Reverb.com deploys must be logged and graphed. Scaling the team and the business should become easier if the teams have quick, repeatable processes that allow them to push new features and bug fixes quickly onto the new DevOps platform. This should allow for a scalable and dynamic workflow for the Reverb.com marketplace so it can be nimble, adapting to the users’ needs as they arise.
Reverb.com’s Reasoning: An Automation Proclamation
Reverb.com made its move to DevOps on the basis that the more the automation, the less the likelihood of human error. From that, Reverb.com development staff should create fewer issues for its customer base as consumers browse the website, make purchases, and sell hardware.
“It’s extremely important to look at root causes here: if we only treat the symptoms every time the site slows or goes down, we will be playing Whac-a-Mole forever,” says Enger. By using the DevOps mindset, initiating repeatable and automated deploys, and measuring performance and application metrics in rea-time, Reverb.com developers should be able to trace when something was released, how it affected users, and whether customers are having a poor experience as a result.
In the hopes that this would equip them to quickly pinpoint issues and fix root causes, the band of coders at Reverb.com set out on the road to DevOps, fired off its first continuous delivery power chord, and awaited feedback.
Tune in next time when DevOps.com reveals how Reverb.com adopted DevOps and used it to meet these development challenges and goals in the exciting conclusion of “A DevOps Metamorphosis Echoes Through Reverb.com”.