DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv Video Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv Video Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB

Home » Blogs » DevOps Practice » Fast Data is Forcing New Strategies in DevOps

Fast Data is Forcing New Strategies in DevOps

Avatar photoBy: Karthik Ramasamy on October 22, 2019 1 Comment

DevOps has established itself as a critically important discipline, developing and leveraging unique tooling and processes to deliver many types of applications more rapidly, efficiently and reliably. As data has become a more central component in these applications, DataOps has emerged to work hand-in-hand with DevOps to better support the role of data in the modern enterprise.

Related Posts
  • Fast Data is Forcing New Strategies in DevOps
  • Are We Leaving Developers Out of DevOps Spinoffs?
  • Everything Ops: Are DevOps Principles Being Applied Too Broadly?
    Related Categories
  • Blogs
  • DevOps Practice
  • Enterprise DevOps
    Related Topics
  • data operations
  • DataOps
  • fast data
Show more
Show less

Yet, a new generation of applications–such as real-time, data streaming and microservices applications, including a growing share that run natively in and across the cloud and datacenter–is already threatening to upend established practices and tools that have been the foundation of DataOps and DevOps collaboration. DevOps is built around an agile mindset, but what must the discipline itself do to adapt?

TechStrong Con 2023Sponsorships Available

These new applications are born of the concept of fast data: acting and reacting to data immediately as it hits the enterprise. In contrast to classic big-data applications that gather, aggregate, store and then later process information–typically in batches–fast data focuses on immediacy to transform, refine and analyze data as it arrives, enabling users, applications and services to quickly act on that information. In many instances this is in support of a larger breed of intelligent applications expected to act on the data without requiring any sort of human intervention.

The organizational benefits come in the form of speed and improved insight. While the former might be obvious, the latter is no less important: Knowing and acting on what’s happening right now significantly improves accuracy, nimbleness and responsiveness.

For DevOps, success in this new era of fast data applications is not possible by applying existing practices and solutions. It requires an evolution, both in those tools/practices themselves, and in how teams approach and think about the overall process of being part of a data-driven organization.

That’s because a number of things change in the world of fast data:

  • Applications are no longer self-contained but comprised of many interdependent components that may stretch across geographies, infrastructure and disciplines.
  • Data flows are the critical link between these components.
  • Data is being collected, processed, analyzed and acted upon before it reaches traditional repositories such as data lakes or data warehouses (if it ever reaches them at all).
  • Fast data applications in production must always be on to keep up with the flow of data, and must frequently handle significant scaling challenges, both up and down.

As an example of these new considerations, contemplate a classic scenario that might consist of a database and content application, with middle-tier logic in between. A typical DevOps process might involve an update to that mid-tier logic that is then deployed, tested and iterated.

In the realm of fast data, you might instead have eight different data services that talk to a dozen different components, all of which communicate with yet more downstream processing applications. Deploying any changes in isolation without an understanding of the various data service and application interfaces might cause the whole system to fall apart. Working in the world of fast data requires the DevOps team to work with DataOps to validate that individual components can be updated and deployed without harm, or establish how to deploy the whole system or portions of the system as a set. These are considerations that weren’t in play when the number of interface points was relatively small and relied on standard protocols or processes.

Fortunately, there are steps DevOps can take to successfully enable these new fast data applications, including:

  • Recognize the necessity of delivering streaming/real time/fast data infrastructure not only for the production environment, but also for the development environment. Most of development and testing for monolithic batch applications can be done using basic infrastructure. However, development and testing of fast data applications requires infrastructure that can stream and process data on-demand. DevOps teams need to deploy data infrastructure that can easily support that.
  • Work with DataOps professionals and/or principles to implement technologies and processes that make it easy to move fast data applications from testbeds into full production environments. Fast data applications are typically composed of a set of services, so moving an application into production requires a carefully coordinated and orchestrated process. DevOps needs to make that process as simple and reliable for developers as possible.
  • Support an always-on environment. Fast data applications by definition need to act on data as soon as it arrives. That means that maintenance downtime for upgrades or reconfiguration is not allowed. DataOps teams need to identify and choose technologies that are designed to operate without interruption or disruptive degradation not only in the face of failures but also in the face of day-to-day maintenance and operations.
  • Design for the cloud from the start. Fast data applications are typically deployed all or part in the cloud, collecting, acting on and distributing data inside and outside the enterprise. That makes it critical for DevOps teams to choose fast data technologies that offer cloud-native resiliency, scalability (up and down), high performance and ease of configuration/operation.

As mentioned earlier, it’s also important DevOps teams adjust their perspective toward data flows supported by infrastructure, rather than just considering application components.

Holistically, data is flowing through a series of interconnected components toward an outcome; the components themselves are just transitional steps in that journey. Changes to any given component must be framed in terms of implications both upstream and down. If a component is changed, does the upstream source need to adjust accordingly? Does the adjustment impact downstream users/applications that rely on the data flow? In larger scenarios this upstream/downstream journey might entail multiple hoops to different systems and business processes. It’s no longer possible to approach changes with a break/fix mentality of a few interconnected components, it must instead consider the deployment or revision to an entire data flow that might rely on a large number of microservices.

Fast data applications—from retail and manufacturing to financial services, media/entertainment and more—aren’t just coming in the future, they’re already here and are enabling a wealth of new business opportunities. DevOps teams need to adjust their practices and tools to best accommodate this new paradigm that’s equally about the data flow and the applications. And they can start preparing for this new fast data landscape by asking a single question: What could I do if I knew what the customers, partners and systems were doing right now?

— Karthik Ramasamy

Filed Under: Blogs, DevOps Practice, Enterprise DevOps Tagged With: data operations, DataOps, fast data

« DevOps Chat: Service Mesh Tracing, from Envoy, Omnition to Splunk
Extreme Networks Unveils ExtremeCloud IQ Cloud Management Application »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Evolution of Transactional Databases
Monday, January 30, 2023 - 3:00 pm EST
Moving Beyond SBOMs to Secure the Software Supply Chain
Tuesday, January 31, 2023 - 11:00 am EST
Achieving Complete Visibility in IT Operations, Analytics, and Security
Wednesday, February 1, 2023 - 11:00 am EST

Sponsored Content

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Practical Approaches to Long-Term Cloud-Native Security

December 5, 2019 | Chris Tozzi

Latest from DevOps.com

Stream Big, Think Bigger: Analyze Streaming Data at Scale
January 27, 2023 | Julia Brouillette
What’s Ahead for the Future of Data Streaming?
January 27, 2023 | Danica Fine
The Strategic Product Backlog: Lead, Follow, Watch and Explore
January 26, 2023 | Chad Sands
Atlassian Extends Automation Framework’s Reach
January 26, 2023 | Mike Vizard
Software Supply Chain Security Debt is Increasing: Here’s How To Pay It Off
January 26, 2023 | Bill Doerrfeld

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

GET THE TOP STORIES OF THE WEEK

Most Read on DevOps.com

What DevOps Needs to Know About ChatGPT
January 24, 2023 | John Willis
Microsoft Outage Outrage: Was it BGP or DNS?
January 25, 2023 | Richi Jennings
Five Great DevOps Job Opportunities
January 23, 2023 | Mike Vizard
Optimizing Cloud Costs for DevOps With AI-Assisted Orchestra...
January 24, 2023 | Marc Hornbeek
A DevSecOps Process for Node.js Projects
January 23, 2023 | Gilad David Maayan
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.