DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • DevOps Chats
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Rocket on DevOps.com
    • Traceable on DevOps.com
    • Quali on DevOps.com
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Digital Anarchist
  • Media Kit
  • About
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • Leadership Suite
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More Topics
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » What Happens to Test in a DevOps World?

What Happens to Test in a DevOps World?

By: contributor on June 13, 2016 1 Comment

I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use.  However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.

Recent Posts By contributor
  • How to Ensure DevOps Success in a Distributed Network Environment
  • Dissecting the Role of QA Engineers and Developers in Functional Testing
  • DevOps Primer: Using Vagrant with AWS
More from contributor
Related Posts
  • What Happens to Test in a DevOps World?
  • DevOps: Shift Left to Reduce Failure
  • What is MLOps? DataOps? And Why do They Matter?
    Related Categories
  • Blogs
  • Doin' DevOps
    Related Topics
  • Automated testing
  • devops
  • production testing
  • security testing
  • shift left
  • shift right
  • test
  • testing
Show more
Show less

Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.

DevOps/Cloud-Native Live! Boston

Shift Left

DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.

Benefits of left-shifting test include:

  • Move testers into scrum teams to work on stories together with developers.
  • Design tests in parallel with development.
  • Give developers and testers the flexibility to help each other to reduce bottlenecks.
  • Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
  • Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
  • Virtual elimination of the test bottleneck that occurs after development is completed.

Shift Right

The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.

Benefits of testing in production include:

  • Better understanding of the real production environment and workloads.
  • Early feedback from customers on new features and user experience.
  • Better resiliency.
  • Overall better user experience for customers.

No Shift

Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release.  These tests include:

  • Performance testing
  • End-to-end use-case testing
  • Security testing (don’t think about doing this in production!)
  • Manual testing – often for user interface or mobile testing that is most difficult to automate

DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.

About the Author/Joan Wrabetz

Joan's image croppedJoan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.

Filed Under: Blogs, Doin' DevOps Tagged With: Automated testing, devops, production testing, security testing, shift left, shift right, test, testing

Sponsored Content
Featured eBook
The 101 of Continuous Software Delivery

The 101 of Continuous Software Delivery

Now, more than ever, companies who rapidly react to changing market conditions and customer behavior will have a competitive edge.  Innovation-driven response is successful not only when a company has new ideas, but also when the software needed to implement them is delivered quickly. Companies who have weathered recent events ... Read More
« Building to Scale
What Can Digital Transformation Do For You? »

TechStrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Modernizing Jenkins Pipelines With CD Automation
Tuesday, May 17, 2022 - 11:00 am EDT
Applying the 2022 OSSRA Findings to Software Supply Chain Risk Management
Tuesday, May 17, 2022 - 1:00 pm EDT
Getting Mainframe and IBM i Data to Snowflake
Tuesday, May 17, 2022 - 3:00 pm EDT

Latest from DevOps.com

Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil
May 16, 2022 | Vladi Sandler
Why Data Lineage Matters and Why it’s so Challenging
May 16, 2022 | Alex Morozov
15 Ways Software Becomes a Cyberthreat
May 13, 2022 | Anas Baig
Top 3 Requirements for Next-Gen ML Tools
May 13, 2022 | Jervis Hui
Progress Expands Scope of Compliance-as-Code Capabilities
May 12, 2022 | Mike Vizard

Get The Top Stories of the Week

  • View DevOps.com Privacy Policy
  • This field is for validation purposes and should be left unchanged.

Download Free eBook

DevOps: Mastering the Human Element
DevOps: Mastering the Human Element

Most Read on DevOps.com

Agile/Scrum is a Failure – Here’s Why
May 10, 2022 | Richi Jennings
How Waterfall Methodologies Stifle Enterprise Agility
May 12, 2022 | Jordy Dekker
How to Secure CI/CD Pipelines With DevSecOps
May 11, 2022 | Ramiro Algozino
Update Those Ops Tools, Too
May 11, 2022 | Don Macvittie
Progress Expands Scope of Compliance-as-Code Capabilities
May 12, 2022 | Mike Vizard

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2022 ·Techstrong Group, Inc.All rights reserved.