DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • DevOps Chats
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Rocket on DevOps.com
    • Traceable on DevOps.com
    • Quali on DevOps.com
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Digital Anarchist
  • Media Kit
  • About
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • DevOps Onramp
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » DevOps in the Cloud » Lessons Learned From the Salesforce Outage

Lessons Learned From the Salesforce Outage

By: Robert Reeves on June 17, 2019 Leave a Comment

Everyone has bad days. Every company has been through some kind of outage due to a buggy database deployment. Even the best of the best, with highly trained staff, world-class best practices and well-thought-out processes make mistakes. On May 17, Salesforce.com had a bad day.

Recent Posts By Robert Reeves
  • Back to DevOps School: What Tools are in Your Backpack?
  • DevOps: The (Absolutely Critical) Cloud Enabler
More from Robert Reeves
Related Posts
  • Lessons Learned From the Salesforce Outage
  • The Rogers Outage of 2022: Takeaways for SREs
  • Busting 5 Common Database Modernization Myths
    Related Categories
  • Blogs
  • DevOps in the Cloud
    Related Topics
  • customer experience
  • database
  • Salesforce outage
  • software update
  • system of record
Show more
Show less

What Happened?

The company deployed a faulty database change script that broke permission settings in production and gave users read and write access to restricted data. This opened the door for an unauthorized employee to steal or tamper with the company’s data. As a result, Salesforce needed to take large parts of its infrastructure down to find and properly fix the issue. The outage lasted 15 hours, 8 minutes. According to Gartner’s cost-of-downtime formula ($5,600/minute), this outage cost approximately $5 million. Plus, since so many companies rely on Salesforce, it was a very visible and embarrassing outage. (Just take a look at #SalesforceDown and #permissiongeddon on social media.)

AppSec/API Security 2022

Salesforce had to shut everything down because of the way databases work. It’s not as easy as pulling a single application. Who knows how many Salesforce employees worked like mad to take the whole database down, find the offending database script, and restore everything—all because of one change script. That’s not a fun way to spend a weekend.

What Does This Tell Us?

Historically, Salesforce customers have experienced very little disruption in service. On the day of the outage, many loyal customers were tweeting about how rock solid the service has been, and that’s impressive.

That being said, this outage should be a wake-up call for users to realize their dependency on their platform, which has become a more integral part of how we conduct business. I’ve heard anecdotes of entire offices being unable to complete work that Friday.

The customer reactions show that they clearly have their stuff together over at Salesforce. What this outage shows is less about any shortcomings of this company specifically; but rather that everyone has blind spots, no matter how robust the testing process is.

Lessons for IT professionals:

  • Don’t forget the database. Unfortunately, this problem is all too common. There are countless cases of a database change script being executed in production and causing unexpected issues. This indicates that the database is often the forgotten part of the software test and release cycle. Unless companies start to seriously consider the database when it comes to change management, things like this will continue to happen.
  • Automate everything. Manual efforts fail. Humans make mistakes. Companies such as Salesforce that handle sensitive customer data should not put it on the internet unless they have automated every aspect of the system, including the database. Companies need a robust DataOps process, one that includes production-like data and automates the validation database changes.

Lessons for end users:

  • Establish a continuity plan. Organizations that depend on systems such as Salesforce need to have a business continuity plan to continue operations if they go down. This includes syncing Salesforce tasks and calendar with Office365, Exchange or Google Apps. However, since Salesforce has become the system of record for most companies that use the technology, having a backup of the data will be very difficult—after all, the point of a company using the platform in the first place was to avoid hosting its own data.
  • Demand accountability. Additionally, users can and should demand more accountability from software vendors. This can take the form of demanding a refund, but also customers need to speak to their account manager to understand what steps Salesforce is doing to make sure this does not happen in the future. It’s 2019 and companies update software all the time without taking down their production systems at the end of a quarter. Moreover, companies also can demand that companies don’t change production systems, outside of necessary security issues, during business hours.

The bottom line is that, while the mistakes that led to the Salesforce outage are very costly and highly visible to customers, they are also entirely preventable.

— Robert Reeves

Filed Under: Blogs, DevOps in the Cloud Tagged With: customer experience, database, Salesforce outage, software update, system of record

Sponsored Content
Featured eBook
The 101 of Continuous Software Delivery

The 101 of Continuous Software Delivery

Now, more than ever, companies who rapidly react to changing market conditions and customer behavior will have a competitive edge.  Innovation-driven response is successful not only when a company has new ideas, but also when the software needed to implement them is delivered quickly. Companies who have weathered recent events ... Read More
« The Value of Visual Data in Decision-Making
Moogsoft Advances AIOps Workflow »

TechStrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Transforming the Database: Critical Innovations for Performance at Scale
Tuesday, August 23, 2022 - 1:00 pm EDT
Modern Data Protection With Metallic DMaaS: Hybrid, Kubernetes and Beyond
Wednesday, August 24, 2022 - 11:00 am EDT
DevOps Unbound: Report on AI-Augmented DevOps
Tuesday, August 30, 2022 - 11:00 am EDT

Latest from DevOps.com

Agile Sucks (Redux) | Plus: DevOps on Mars
August 18, 2022 | Richi Jennings
Survey Shows Steady DevSecOps Progress
August 18, 2022 | Mike Vizard
Why DevOps Teams Need Security Engineers
August 18, 2022 | Mike Vizard
Time-Series Database Basics
August 18, 2022 | Jeff Tao
Busting 5 Common Database Modernization Myths
August 18, 2022 | Anthony Loss

GET THE TOP STORIES OF THE WEEK

Download Free eBook

The 101 of Continuous Software Delivery
New call-to-action

Most Read on DevOps.com

Next-Level Tech: DevOps Meets CSOps
August 12, 2022 | Jonathan Rende
The Benefits of a Distributed Cloud
August 12, 2022 | Jonathan Seelig
Techstrong TV: Scratching the Surface of Testing Through AI
August 12, 2022 | Alan Shimel
Building a Platform for DevOps Evolution, Part One
August 16, 2022 | Bob Davis
5 Ways to Prevent an Outage
August 15, 2022 | Ashley Stirrup

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2022 ·Techstrong Group, Inc.All rights reserved.