DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • Calendar View
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • Calendar View
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Cloud Native Now
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • CI/CD
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Sustainability
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Chronosphere Adds Professional Services to Jumpstart Observability
  • Friend or Foe? ChatGPT's Impact on Open Source Software
  • VMware Streamlines IT Management via Cloud Foundation Update
  • Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
  • No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs

Home » Blogs » Persistent Storage: You’ll be Hearing More

Persistent Storage: You’ll be Hearing More

Avatar photoBy: Don Macvittie on December 19, 2017 Leave a Comment

As containers take over the IT environment, a problem that has existed since virtual machines (VMs) became popular has become more urgent: how to keep persistent storage, and keep it consistent.

Recent Posts By Don Macvittie
  • Forget Change, Embrace Stability
  • Is AI the Answer to Devs’ Documentation Problem?
  • Standardize: It’s Not the Where. Sometimes it’s Not the What
Avatar photo More from Don Macvittie
Related Posts
  • Persistent Storage: You’ll be Hearing More
  • Breaking Up and Making Up: Composable Persistent Storage for Kubernetes
  • MapR Accelerates Separation of Compute and Storage
    Related Categories
  • Blogs
  • Containers
  • DevOps in the Cloud
    Related Topics
  • containers
  • Persistent Storage
  • transactional
Show more
Show less

Just as a business designed for foot traffic of 50 to 100 people per hour does not work well when it sees 500 to 1,000 people per hour (think Black Friday in most places these days), storage designed to hold data that persists across instances and even installations often struggles in containerized environments.

There are a couple of reasons why this is so.

First, we now treat our servers as disposable resources. If one dies, we leave it dead and simply spin up another. Normal processing keeps the dead server for later evaluation, but many shops do not follow this best practice. The issue has to do with servers that die when writing to persistent storage—what was the state of the write? Did it get interrupted mid-transfer? Luckily, applications can be designed to account for this, and the increasing prevalence of transactional processing in applications that are not based on a database management system (DBMS) shows that people are doing just that.

Second, the number of server instances hitting those persistent stores is higher in a containerized environment—they are smaller, lighter and faster to spin up and their container density is generally higher than VM density. Contention issues go from zero to full-on when an app goes from “one server is writing to this persistent store” to “two servers are writing to this persistent store.” And often there are hundreds of servers writing to the persistent store. As mentioned, the stop-gap is to implement some form of transactional processing to prevent writes and overwrites.

Persistent Storage: It’s Got Issues

The problem is, we end up with transactional processing written into dozens or hundreds of server instances to protect a small number of persistent disks. This is not optimal, and at a minimum increases software complexity.

That’s why you’ll be hearing more. Here are a few places working on simplified but powerful persistence ideas:

{code} Rex Ray

Originally part of EMC, {code} is Dell’s open source group and it has led the charge on the container storage interface (CSI) that facilitates persistent storage. Examples the company has implemented include databases and Minecraft, but the technology works for anything that must be stored on disk.

Container Orchestration Engines and Cloud Providers

All of the well-known container orchestration engines (COE) and infrastructure-as-a-service (IaaS) cloud vendors have methods to handle persistent disk—some more transactional than others and some more work than others. Expect this process to become standardized (and possibly for that standardization to take the form of CSI above) in the next year or so. If persistent data is critical to your container environment, it is worth looking at how each solves the problem, as some of the solutions are procedural (such as Docker’s recommendation), and thus are somewhat portable across COE solutions. The weakness of IaaS providers is portability between systems, but that is true of all IaaS procedures/solutions at this time.

Virtualization Engines

Both VMware vSAN and Microsoft NTFS have systems for providing persistent disk. While much of this is the simple case of “disk that doesn’t disappear when the container does,” that is the bulk of persistence needs. Expect the support to continue to expand, perhaps even going so far as to implement 100 percent transactional shares.

Storage Array Vendors

Storage array vendors have useful tools for this area. While what you use to access their storage is not too terribly large a concern to these vendors, they do have a vested interest in seeing that their storage meets the needs of the changing enterprise. As such, they are delving into methods to help make persistent storage available and consistent. This is not a huge stretch for the vendors, as the systems they produced were always intended for multiple access. Where they lag (and where Rex Ray steps up) is in interoperability/interchangeability of vendor solutions.

Conclusion

For those with controlled access to persistent/shared storage, these issues are not that big a deal. For high-volume or high-turnover DevOps installations, though, persistent storage is another link in the chain of reliability that must be protected. Keep kicking it, and hopefully before your need becomes urgent, a solid system will be in place to manage and protect persistent stores.

— Don Macvittie

Filed Under: Blogs, Containers, DevOps in the Cloud Tagged With: containers, Persistent Storage, transactional

« The Executive’s Guide to Microservices Chapter 2: The Business Challenge
5 Biggest Ways to Fail at DevOps »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Securing Your Software Supply Chain with JFrog and AWS
Tuesday, June 6, 2023 - 1:00 pm EDT
Maximize IT Operations Observability with IBM i Within Splunk
Wednesday, June 7, 2023 - 1:00 pm EDT
Secure Your Container Workloads in Build-Time with Snyk and AWS
Wednesday, June 7, 2023 - 3:00 pm EDT

GET THE TOP STORIES OF THE WEEK

Sponsored Content

PlatformCon 2023: This Year’s Hottest Platform Engineering Event

May 30, 2023 | Karolina Junčytė

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Latest from DevOps.com

Chronosphere Adds Professional Services to Jumpstart Observability
June 2, 2023 | Mike Vizard
Friend or Foe? ChatGPT’s Impact on Open Source Software
June 2, 2023 | Javier Perez
VMware Streamlines IT Management via Cloud Foundation Update
June 2, 2023 | Mike Vizard
Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
June 2, 2023 | Marc Hornbeek
No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
June 1, 2023 | Richi Jennings

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Most Read on DevOps.com

What Is a Cloud Operations Engineer?
May 30, 2023 | Gilad David Maayan
Forget Change, Embrace Stability
May 31, 2023 | Don Macvittie
Five Great DevOps Job Opportunities
May 30, 2023 | Mike Vizard
No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
June 1, 2023 | Richi Jennings
Checkmarx Brings Generative AI to SAST and IaC Security Tools
May 31, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.