DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • Calendar View
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • Calendar View
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Cloud Native Now
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • CI/CD
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Sustainability
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Dev Jobs are Dead: ‘Everyone’s a Programmer’ With AI ¦ Intel VPUs
  • Logz.io Taps AI to Surface Incident Response Recommendations
  • Why You Need a Multi-Cloud and Multi-Region Deployment Strategy
  • Cloud Drift Detection With Policy-as-Code
  • Checkmarx Brings Generative AI to SAST and IaC Security Tools

Home » Blogs » Enterprise DevOps » Log Analysis, or Log Hoarding

Log Analysis, or Log Hoarding

Avatar photoBy: Chris Riley on December 5, 2014 1 Comment

Let’s take a stroll down memory lane to remember the beginning of web application era:  when a client had a problem, it called the support team to complain about the service failure or bug, and a customer support team would have written an incident report and forward it to the operations team, and then the operations team would examine it, mainly by looking at the logs with whatever helpful information they had from the customer.

Recent Posts By Chris Riley
  • Using Incident Response for Continuous Testing
  • What Is Resilience Engineering?
  • Moving from NOC to the SRE Model
Avatar photo More from Chris Riley
Related Posts
  • Log Analysis, or Log Hoarding
  • Key customer verticals adopt XpoLog’s Augmented Search for IT log analysis
  • Business Value of Log Analysis
    Related Categories
  • Blogs
  • Enterprise DevOps
    Related Topics
  • log analysis
  • system logging
Show more
Show less

TechStrong Con 2023Sponsorships Available

You need Log Analysis

Can you imagine end users to be counted on to report problems today, when the average log today amasses 864,000 entries/day (600 entries/minute) and a network  with 15 devices generates 13 million events per day ? How many people and how much time will be needed to sift through all those files?

The two extreme positions regarding this high volume of data are: “Who needs it? Let’s make some space!”  and “I want to keep all the data, all the time, because having more data equals better decisions and who knows when I will need it!” (digital hoarding).

Probably the right approach is something in the middle: to carefully consider which data are likely to be of used (and are required for PCI-DSS, HIPAA, SOX, GLBA, FISMA compliance audits), and focus only on these data, but for big companies, that own 80% of all that data right now, it doesn’t help too much, because even after all this “clean-up” the amount is still extremely high.

And here comes tools like Splunk to help,  which makes searchable data not only from any  network traffic, Web servers, custom applications, application servers, GPS systems, stock market feeds, social media, but from preexisting structured databases, logs, config files, messages, alerts, scripts and metrics too.

Splunk  offers DevOps’s team members a centralized view across all of the machine data, enabling threat prevention and detection and intelligence gathering, while in the same time it captures web interactions and key metrics such as time spent on page, bounce rates, navigation paths and product performance.

But even if Splunk has essentially become a verb in the world of machine data and is arguably the most feature rich logging solution, it has it’s caveats: beside the fact that its price is prohibitive for small companies, it comes with a steep learning curve and it’s not very user friendly. Setting up Splunk is a lengthy and tedious process, it is highly unlikely that when data is imported for the first time, it would be indexed correctly by Splunk, so you need a team to go through a time-consuming procedure, formatting the coarse data as required, as well as preparing the platform to accurately read each field.

To grapple with complex real life technical issues, not only do you need to have a very good understanding of the entire system and good intuition on what to query,  but also knowledge of a language called Splunk’s Search Processing Language(SPL), which is very powerful but complex.

Every minute the DevOps team spends looking for an issue (performance, bug, security problem etc.) it’s time that the customers are being affected by it, so in the last years the requirements to be front-runner have changed. To react in hours or even minutes from an unexpected and unwanted event happening is not acceptable. You can no longer be only reactive, you need to be proactive in preventing issues. Real-time messages will alert to activities or key metrics, that indicate abnormal system behavior patterns, and allow you to predict and take action before a problem could happen.

Lets contrast the Splunk approach with the “more modern” examples hitting the market.

The modern log analysis platform like Logentries, a cloud based log management and real-time analytics service,  is doing that by creating a future picture of what performance will look like based on historical activity. If the performance or characteristics do not meet this model, admin are alerted immediately.

Through it’s visualization platform, Logentries, it’s aggregating data in real time, and asynchronously pushing information when needed, rather than based on a time-based regular snapshot. In this data are included, not only events that should happen, but also missing events (using the Inactivity Alerting feature): credit cards that are not being processed or if website traffic halts unexpectedly. All that without spending a fortune or hiring an army of experts.

Providing a simple, intuitive and flexible search, Logentries fetches events using keywords, regular expressions and field patterns. Logs are supplemented with real-time information coming from the infrastructure: CPU, memory, and disk I/O information. Using Logentries in combination with HipChat, PagerDuty and Campfire,  it provides actionable insights in every phase of operations, from development to deployment, to ongoing management and support.

The modern log system is designed for Devops  methodology, where collaboration is paramount:

  • the team Annotation feature, allows placing annotation with comments, advice, solution and relevant system context, that can be  seen, by other Devops Engineers, when similar problems are identified later
  • the Shareable dashboards centralizes monitoring by publishing log and time-stamped data and trends across the DevOps team and the entire organization
  • the Notifications module sends warnings to people, groups or the entire team, using custom tagging and real-time alerting, to avoid issues caused by app crashes, memory shortages, request timeouts etc. before they become fatal.

By adding logs from firewalls, routers, vulnerability scanners, Intrusion Detection Systems/Intrusion Prevention Systems (IDS/IPS), modern log analysis  monitors, explores, and diagnoses system security events in real-time and track malicious attempts against the network.

These services can be connected to any platform using an API. They also already have  a wide variety of integrations into popular cloud providers (such as AWS) and has agents and hooks for OS platforms.

Logging  can be improved even more, going beyond making sense of an overwhelming volume of data and providing real time information. New and interesting features appear all the time, let’s take an example of one of them that caught my eye  while researching for this article: passing complete information directly from the production machines logs to the development tools. Because in a production environment the verbosity of logs is often limited by performance constraints, a private company called  Takipi, came with the idea to get from a log file error directly into a recorded debugging session, and as a result seeing source code and variable values at the moment of error.

With the Internet of Things slowly emerging and creating limitless possibilities  of connecting to everything, the volume of data that need to be logged and analyzed will just continue to expand into Brontobytes and Geopbyte, requiring new technologies difficult to predict. Which means the effort of getting value from these logs is critical. Thus the approach of hoard all logs, and learn complex query languages cannot work.

Ten or twenty years from now we will look back and smile thinking of the “legacy” logging software, of the first decade of the century, when the biggest data centre (the Utah Data Centre), was only capable of storing 12 Exabyte of data.

Filed Under: Blogs, Enterprise DevOps Tagged With: log analysis, system logging

« DevOps Sometimes Means Kill It Before Fixing It
10 Top Tips for DevOps Cultural Change »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Securing Your Software Supply Chain with JFrog and AWS
Tuesday, June 6, 2023 - 1:00 pm EDT
Maximize IT Operations Observability with IBM i Within Splunk
Wednesday, June 7, 2023 - 1:00 pm EDT
Secure Your Container Workloads in Build-Time with Snyk and AWS
Wednesday, June 7, 2023 - 3:00 pm EDT

GET THE TOP STORIES OF THE WEEK

Sponsored Content

PlatformCon 2023: This Year’s Hottest Platform Engineering Event

May 30, 2023 | Karolina Junčytė

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Latest from DevOps.com

Dev Jobs are Dead: ‘Everyone’s a Programmer’ With AI ¦ Intel VPUs
June 1, 2023 | Richi Jennings
Logz.io Taps AI to Surface Incident Response Recommendations
June 1, 2023 | Mike Vizard
Why You Need a Multi-Cloud and Multi-Region Deployment Strategy
June 1, 2023 | Jesse Martin
Cloud Drift Detection With Policy-as-Code
June 1, 2023 | Joydip Kanjilal
Checkmarx Brings Generative AI to SAST and IaC Security Tools
May 31, 2023 | Mike Vizard

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Most Read on DevOps.com

CDF Marries Emporous Repository to Ortelius Management Platform
May 26, 2023 | Mike Vizard
Is Your Monitoring Strategy Scalable?
May 26, 2023 | Yoni Farin
The Metrics Disconnect Between Developers and IT Leaders
May 25, 2023 | Mike Vizard
GitLab Adds More AI and Cybersecurity Capabilities to CI/CD Platform
May 26, 2023 | Mike Vizard
What Is a Cloud Operations Engineer?
May 30, 2023 | Gilad David Maayan
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.