DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • DevOps Chats
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Rocket on DevOps.com
    • Traceable on DevOps.com
    • Quali on DevOps.com
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Digital Anarchist
  • Media Kit
  • About
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • Leadership Suite
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More Topics
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » Doin' DevOps » Metrics for DevOps

Metrics for DevOps

Metrics for DevOps

By: Chris Riley on January 26, 2015 9 Comments

Your organization is now committed to DevOps methodology.  You’ve integrated development, operations, and QA, and you have moved drastically away from waterfall and into a fire hose of application development.  How do you find out how well it works?  How do you know if it’s working at all? Many net new DevOps organizations are surprised to find that it was easier to start then they thought, but hard to keep alive. This can be because of menacing habits, or just because the team is too far into the weeds. In a word, you need metrics

Recent Posts By Chris Riley
  • Using Incident Response for Continuous Testing
  • What Is Resilience Engineering?
  • Moving from NOC to the SRE Model
More from Chris Riley
Related Posts
  • Metrics for DevOps
  • Why MTTR is a Vital Metric for DevOps Teams
  • Boosting DevOps Productivity Requires Visibility, Automation
    Related Categories
  • Blogs
  • Doin' DevOps
    Related Topics
  • Business
  • measure
  • plan
Show more
Show less

DevOps/Cloud-Native Live! Boston

You need metrics  not just as a way to measure the success (or lack of success) of your DevOps program, but also as a way to find out how it can be improved, modified, or extended.  Without metrics, you’re flying blind.  With metrics, you have a holistic point of view, you know where you are, where you’re going, where you can go, and how to get there. But I am not talking about analytics tools which measure the activities withing the pipeline. I’m talking about measuring the pipeline itself.

What kinds of factors do DevOps metrics measure?  For the most part, they measure such things as the speed of development, deployment, and customer response, frequency of deployments and failures, repair time, volume of repair request, and the rate of change in these indicators. The focus of DevOps metrics tends to be on deployment, operations, and support (as opposed to design and early-stage development, for example), since most of the ongoing effort associated with DevOps is in these areas.

Most DevOps metrics fall into three general categories:

  • People.  People are an intrinsic part of any DevOps process.  People-oriented metrics measure such things as turnover, capability, and response time. Always start with people. They are the hardest element of any element, and their influence is sometimes hard to spot.
  • Process.  In some ways, DevOps is all about process  —  the continual deployment/operations/support cycle is an ongoing suite of interwoven processes.  But some metrics are more clearly process-oriented than others, particularly those involving continuous delivery, response, and repair.  Development-to-deployment lead time, for example, is a largely process-oriented metric, as are deployment frequency and response time.  Process metrics can be a measure of speed (Where are the bottlenecks, and is the process itself a bottleneck?), appropriateness (Are all steps relevant?), effectiveness (Does it get the job done?), or efficiency (Are the steps in the optimum sequence?  is there a smooth flow within the process?).
  • Technology.  Technology metrics also play a major role in DevOps, measuring such things as uptime (What percentage of the time is the system running?  What about the network, and support applications?) and failure rate (What is the percentage of failed deployments, changes, or units?).

Of course, many DevOps metrics involve all three categories to a greater or lesser degree.  Perhaps the easiest way to see how metrics play out in practice is to look at the key metrics used by Puppet Labs (Puppet is not a customer of mine, nor am I a customer of theirs) :

Deployment (or Change) Frequency

DevOps practices make frequent or continuous deployment possible; large, high-traffic web sites and cloud-based services make it a necessity.  With fast feedback and small-batch development, updated software can be deployed every few days, or even several times per day.  In a DevOps environment, deployment frequency can be a direct or indirect measure of response time, team cohesiveness, developer capabilities, development tool effectiveness, and overall DevOps team efficiency.

Change Lead Time

The time from the start of a development cycle (the first new code) to deployment is the change lead time.  It’s a measure of the efficiency of the development process, of the complexity of the code and the development systems, and also (like deployment frequency) of team and developer capabilities.  If the change lead time is too long, it may be an indication that the development/deployment process is inefficient in certain stages, or that it includes performance bottlenecks.

Change Failure Rate

One of the main goals of DevOps is to turn rapid, frequent deployments into an everyday affair.  Needless to say, in order for such deployments to have value, the failure rate must be low.  It should, in fact, decrease over time, as the experience and capabilities of the DevOps teams increase.  An increasing failure rate, or one that is high and does not go down over time, is a good indication of problems in the overall DevOps process.

Mean Time To Recover (MTTR)

This is the time from a failure to recovery from that failure.  It’s generally a good measure of team capabilities, and like the failure rate, it should show an overall decrease over time (allowing for occasional longer recovery times when the team encounters a technically unfamiliar problem).  MTTR can also be affected by such things as code (or platform) complexity, the number of new features being implemented, and changes in the operating environment (such as migration to a new cloud server).

Based on these metrics, the ideal DevOps team would produce frequent, rapid deployments with a low (and declining) failure rate and a short (and shrinking) recovery time.  In practice, of course, there may be factors which run counter to these trends  —  less need or opportunity for frequent deployments, for example, or frequent changes in operating conditions or requirements.  In general, however, these metrics do cover some of the most important performance issues in DevOps.

The bottom line is that if you’re using DevOps methodology, you need metrics, and you need good metrics tools.  Without metrics, you won’t have any way of knowing if your DevOps implementation is doing what you want it to do, or if it includes problem areas that require your attention.

In the very least, good DevOps metrics should cover the four areas outlined above (deployment frequency, lead time, failure rate, and time to recovery), along with other key indicators of performance which matter to your organization.  And good DevOps metrics tools should present that information to you in ways which are clear, detailed, accurate, and easily configurable, allowing you to quickly focus on potential problem areas and other key points in the DevOps process.

Tools that measure DevOps performance will typically monitor real-time processes and conditions (such as uptime and downtime, or volume of traffic) and log events (builds, deployments, failures, repair tickets, etc.).  More generalized project-management tools which track factors such as project schedule and goals vs. performance can also provide valuable DevOps metrics.

A note of caution:  when you start to look closely at DevOps metrics, you’ll find that there are quite a lot of them available.  Before you dive into DevOps metrics, it’s important to understand clearly which metrics are most important to you, and how they apply to your DevOps program.  If you don’t know what you’re looking for, it’s too easy to get lost in a maze of not-very-relevant data, but if you stay focused, you can head straight for the DevOps metrics gold, and put it to good use.

Filed Under: Blogs, Doin' DevOps Tagged With: Business, measure, plan

Sponsored Content
Featured eBook
The State of the CI/CD/ARA Market: Convergence

The State of the CI/CD/ARA Market: Convergence

The entire CI/CD/ARA market has been in flux almost since its inception. No sooner did we find a solution to a given problem than a better idea came along. The level of change has been intensified by increasing use, which has driven changes to underlying tools. Changes in infrastructure, such ... Read More
« The role of middleware in DevOps
Google Hangout: InterConnect 2015 preview from IBM & Nationwide on #DevOps »

TechStrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Accelerating Continuous Security With Value Stream Management
Monday, May 23, 2022 - 11:00 am EDT
The Complete Guide to Open Source Licenses 2022
Monday, May 23, 2022 - 3:00 pm EDT
Building a Successful Open Source Program Office
Tuesday, May 24, 2022 - 11:00 am EDT

Latest from DevOps.com

DevSecOps Deluge: Choosing the Right Tools
May 20, 2022 | Gary Robinson
Managing Hardcoded Secrets to Shrink Your Attack Surface 
May 20, 2022 | John Morton
DevOps Institute Releases Upskilling IT 2022 Report 
May 18, 2022 | Natan Solomon
Creating Automated GitHub Bots in Go
May 18, 2022 | Sebastian Spaink
Is Your Future in SaaS? Yes, Except …
May 18, 2022 | Don Macvittie

Get The Top Stories of the Week

  • View DevOps.com Privacy Policy
  • This field is for validation purposes and should be left unchanged.

Download Free eBook

The State of Open Source Vulnerabilities 2020
The State of Open Source Vulnerabilities 2020

Most Read on DevOps.com

Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil
May 16, 2022 | Vladi Sandler
Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Micro...
May 17, 2022 | Richi Jennings
Making DevOps Smoother
May 17, 2022 | Gaurav Belani
DevOps Institute Releases Upskilling IT 2022 Report 
May 18, 2022 | Natan Solomon
Why Data Lineage Matters and Why it’s so Challenging
May 16, 2022 | Alex Morozov

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2022 ·Techstrong Group, Inc.All rights reserved.