DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • DevOps Chats
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Rocket on DevOps.com
    • Traceable on DevOps.com
    • Quali on DevOps.com
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Digital Anarchist
  • Media Kit
  • About
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • Leadership Suite
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » DevOps Practice » The State of Alerting in the IT Ops world

Alerting IT Ops World

The State of Alerting in the IT Ops world

By: Orlee Berlove on January 11, 2018 Leave a Comment

OnPage Corp. just finished a survey of more than 100 ITOps professionals from across the United States. Our goal was to acquire a greater understanding of how well engineers in the industry are performing when it comes to critical alerting and alert management of their IT teams.

Recent Posts By Orlee Berlove
  • The State of IT Ops
More from Orlee Berlove
Related Posts
  • The State of Alerting in the IT Ops world
  • Why MTTR is a Vital Metric for DevOps Teams
  • How Enterprise Architects Can Help IT Operations
    Related Categories
  • Blogs
  • DevOps Practice
    Related Topics
  • alerts
  • IT operations
Show more
Show less

We wanted to understand the antecedents of alert fatigue for ITOps that appear earlier in the food chain. We also wanted to see how many alerts teams receive per day as well as who gets alerted. We wanted to understand how alerts are managed. And, we wanted to see how well teams analyze their actions and take those lessons forward.

DevOps Connect:DevSecOps @ RSAC 2022

In many ways, the survey was successful. We received a large number of responses from a number of industries and acquired a strong sense of how ITOps is performing across the country. Unfortunately, we also saw that for all the Chaos Monkeys and strides toward improved response to alerting, there is still a significant lack of progress.

What’s All the Buzz About?

Automated alerting is an essential component of monitoring. Automated alerts are what allow teams to receive automatically generated alerts from multiple points along their IT stack and software. In theory, this multitude of alerts is what enables teams to more quickly identify the causes of a problem and minimize the severity. The hope is, that with early recognition of the issue, engineers will be able to minimize service degradation and disruption.

But alerts aren’t always as effective as they could be or need to be. Real problems often are lost in a sea of noisy alarms. As our survey showed, this is because teams are inundated with alerts coming in via multiple formats. Moreover, the barrage of alerts leaves teams inundated and practically unable to cope.

The Law of Above Average

Our survey showed that more than 80 percent of IT teams are alerted to critical incidents via email. General best practices would dictate that email is fine for daily communication inside a business. However, for critical incidents, email is less than ideal, as it allows critical incidents to get buried under a pile of other emails. There is no way for critical issues to rise to the top of the pile.

Since our questions were multiple choice, respondents could provide multiple answers with regard to how they received notice of critical incidents. So, while email was the most prevalent form of communication, individuals indicated that they are also simultaneously receiving alerts by SMS or phone call.  Our survey showed that 58.9 percent and 51.4 percent of respondents received alerts via these methods.

Already from this nugget of information, we see the opportunity for both information overload and an opportunity for missed alerts. By simultaneously receiving alerts through multiple formats, the level of irritation and overload inevitably rises. At the same time, if emails are the only form in which IT professionals receive alerts then there is a high opportunity to miss alerts all together.

How Many Alerts Was That?

The survey results also indicated that just over 41 percent of ITOps receive 11 alerts or more per day. Additionally, just over 20 percent of this group received 40 alerts or more per day. While 40 alerts is clearly more than a team can reasonably manage or should manage, this figure also goes a long way toward explaining why some alerts just get missed. If more than 40 alerts are sent to you and your team every day, it becomes very difficult to prioritize alerts and determine which should be handled first.

Perhaps to better manage this large number of alerts, many teams use escalation procedures. Our survey showed that 76.6 percent of respondents have some sort of escalation procedure in place. At the same time, the most frequent ways to escalate critical responses was through email or SMS.

The conclusions one can draw from these numbers are that, despite the large number of papers written on improving alert management, many ITOps have not been able to achieve this end. While our survey did show that just shy of 59 percent receive a manageable number of alerts, 41 percent are inundated.

Not Just Intelligence, Business Intelligence

Perhaps analysts of the industry could be more optimistic if they saw that teams were using analytics to track how well they are performing. If teams employed analytics, they would be better able to review their progress, see where they are failing to meet the grade and then embark on routines to improve. Unfortunately, this is not the case.

When asked whether their team has employed any type of business intelligence to review and analyze their team’s performance, more than 70 percent reported that they did not subscribe to any BI platform. The problem with this result is more than just a missed opportunity; it is also the loss of opportunity to fundamentally improve the business at many levels.

One of the most important reasons why you need to invest in an effective BI system is because such a system can improve efficiency within your organization and, as a result, increase productivity. Effective business intelligence can also improve the decision-making processes at all levels of management and improve your tactical strategic management.

Yet by forgoing investments in these BI tools, teams are failing to investigate their processes and methods that would improve their team and minimize alert fatigue.

A Call for Smart Alerting

The lesson can be drawn from this is that companies don’t necessarily need more alerting. What they do need is to shift toward more smart alerting.

Smart alerting means that not every bump on the monitoring screen gets tied to an alert. Instead, monitoring output is calibrated so that possibilities are aligned with probabilities and impacts. Alerts also get sent to the teams or individuals that are best able to manage the issue. Additionally, alerts are actionable and come with instructions regarding what the problem might be.

Smart alerting also means that teams use business intelligence tools such as reports and graphs and charts to determine which of their practices have been effective or not effective. Without this insight, teams are often unaware of the subtle points that could really impact their team and provide them with a way to improve their output.

Conclusion

There are a number of insights that can be garnered from our survey. I encourage you to take a moment and download a copy of the study and see what you can learn that will help your team.

— Orlee Berlove

Filed Under: Blogs, DevOps Practice Tagged With: alerts, IT operations

Sponsored Content
Featured eBook
The 101 of Continuous Software Delivery

The 101 of Continuous Software Delivery

Now, more than ever, companies who rapidly react to changing market conditions and customer behavior will have a competitive edge.  Innovation-driven response is successful not only when a company has new ideas, but also when the software needed to implement them is delivered quickly. Companies who have weathered recent events ... Read More
« NodeSource Increases Metrics in Node.js Development Platform
Clearlake Capital Acquires Perforce Software »

TechStrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Continuous Deployment
Monday, July 11, 2022 - 1:00 pm EDT
Using External Tables to Store and Query Data on MinIO With SQL Server 2022
Tuesday, July 12, 2022 - 11:00 am EDT
Goldilocks and the 3 Levels of Cardinality: Getting it Just Right
Tuesday, July 12, 2022 - 1:00 pm EDT

Latest from DevOps.com

Rust in Linux 5.20 | Deepfake Hiring Fraud | IBM WFH ‘New Normal’
June 30, 2022 | Richi Jennings
Moving From Lift-and-Shift to Cloud-Native
June 30, 2022 | Alexander Gallagher
The Two Types of Code Vulnerabilities
June 30, 2022 | Casey Bisson
Common RDS Misconfigurations DevSecOps Teams Should Know
June 29, 2022 | Gad Rosenthal
Quick! Define DevSecOps: Let’s Call it Development Security
June 29, 2022 | Don Macvittie

Get The Top Stories of the Week

  • View DevOps.com Privacy Policy
  • This field is for validation purposes and should be left unchanged.

Download Free eBook

The State of the CI/CD/ARA Market: Convergence
https://library.devops.com/the-state-of-the-ci/cd/ara-market

Most Read on DevOps.com

What Is User Acceptance Testing and Why Is it so Important?
June 27, 2022 | Ron Stefanski
Chip-to-Cloud IoT: A Step Toward Web3
June 28, 2022 | Nahla Davies
Rust in Linux 5.20 | Deepfake Hiring Fraud | IBM WFH ‘New No...
June 30, 2022 | Richi Jennings
DevOps Connect: DevSecOps — Building a Modern Cybersecurity ...
June 27, 2022 | Veronica Haggar
Common RDS Misconfigurations DevSecOps Teams Should Know
June 29, 2022 | Gad Rosenthal

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2022 ·Techstrong Group, Inc.All rights reserved.