DevOps.com

Where the world meets DevOps

  • Home
  • Features
  • Neighborhoods
    • Leadership Suite
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • DevOps Practice
    • DevOps Toolbox
    • DevOps Security
    • Container Journal
    • Microservices Journal
    • ROELBOB
  • Webinars
    • Upcoming
    • On-Demand
  • Library
  • Chat
  • News
  • Directory
  • About
  • Connect
  • News Releases
    • Facebook
    • Google+
    • Linkedin
    • Twitter

Home » Features » Loggly Derived Fields Add Structure To Unstructured Logs

Loggly Derived Fields Add Structure To Unstructured Logs

David GeerBy David Geer on July 14, 2015 1 Comment

The Gist of Loggly’s Derived Fields Gesture

 
Recent Posts By David Geer
  • Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 2
  • Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 1
  • Sounding the Death Knell for Agile: Not so Fast!
David Geer More from David Geer
Related Posts
  • Built.io Teams Up with Loggly to Bring More Effective Log Management to Customers
  • Loggly Closes $11.5M of New Financing
  • Logging Wins for Devops and Security
    Related Categories
  • Features
    Related Topics
  • logs
  • structered
Show more
Show less
 

A top log management product vendor, Loggly announced an update to its tooling last month in the form of its new Derived Fields. The vendor has drilled down past the summaries in its Dynamic Field Explorer interface into Derived Fields that use metadata to structure unstructured log data.

 

Where traditional log management and analysis products have typically required you to create one-off regular expressions for custom analyses—expressions that you must build anew each time you want to put log data under a microscope—Loggly now enables you to use rules rooted in regular expressions that you create once to structure unstructured data when your systems first enter logs into the tool. You can look at resulting Derived Fields instantly, every time you need to do log analysis, according to Hector Angulo, Head of Products, Loggly.

 

The necessity for the Derived Fields capability arises from the fact that logs that escape structured creation contain so much untapped value. Until now, Loggly’s approach has been to automatically apply structure to common log types. “But we find that more than one-third of customer logs are still not structured. Even worse, many of these logs were created by developers for their own use, without the expectation that others would need to rely on them for critical troubleshooting. As such, it can be really hard for everyone but the original developer to decipher what specific logs mean,” says Angulo. Derived Fields attempt to retrieve 100-percent of that value from the log data by making analysis possible for any team member.

 

How Loggly Dogs Data Using Derived Fields

 

Loggly absorbs data from common log types such as Apache or JSON. By adding metadata that describes unstructured log data and then “injecting” context and structure into logs or parts of a log, Loggly’s Derived Fields reclaim obscure developer logs. Fields that uniformly describe the same kinds of data elements across logs enable users to compare data within logs to see what is happening across a DevOps environment.

 

Staff members who can create custom parsing rules based on regular expressions can set up the rules the DevOps teams will need at the start. Then, unless you come up with a new rule you want to add, your team members won’t have to create regular expressions again, and certainly not in order to search log data on the other end of that pipe.

 

Using Derived Fields

 

Using Loggly’s Derived Fields, you can drill down on a specific value in a field. Click on the field name and you can inspect all log events that have the same type of value. A summary chart above the individual event log will automatically refresh to summarize the events of the data from the given value, says Angulo.

 

You can easily see other log events that share similar data or characteristics in order to arrive at insights about what is going wrong, or right with the software.

 

How Derived Fields Help DevOps

 

“Rather than starting with an empty search box, Loggly starts off with bird’s-eye-view summaries using Dynamic Field Explorer,” says Angulo. From there, you drill down to fine grain details.

 

The time that Loggly’s Derived Fields save, time that it would normally take to create one-off queries for every analysis, is critical if you’re trying to solve a problem that is preventing end users from completing tasks and generating revenue for the business, says Angulo.

 

According to Angulo, Derived Fields specifically help DevOps teams to:

 

• Resolve issues faster because it is easier to spot the data that matters.

 

• Perform advanced analytics even with legacy applications that send out unstructured, text-based logs that you cannot update or don’t have the bandwidth to update.

 

• Extend advanced log analysis to more team members. You don’t have to be a regex wizard to gain insight from unstructured logs.

 

• Finally, the intuitive, navigable summaries generated in Field Explorer equate to less training time, less time with reference guides, and more time on data analysis and problem solving.

 

A Cautionary Note

 

As with any DevOps tool update, the proof lies in how Derived Fields work in your development process. Loggly touts 5K customers using its existing product. I’m sure they’re not trying to chase anyone away.

 

Are you using this tool? Do you have other input on this topic? See that empty comments field below?

  
Sponsored Content
Featured eBook
Modernizing Your Database Processes with DevSecOps

Modernizing Your Database Processes with DevSecOps

Not long ago, companies enjoyed the luxury of releasing new products and services occasionally. In those days, developers might work for weeks, months, even years, accumulating database changes across a mix of update scripts and shared development databases. In this document, we will describe four pillars to building the infrastructure ... Read More
 

Filed Under: Features Tagged With: logs, structered

  • Click to share on LinkedIn (Opens in new window)
  • Click to share on Twitter (Opens in new window)
  • Click to share on Facebook (Opens in new window)
  • Click to share on Reddit (Opens in new window)
  • Click to share on Google+ (Opens in new window)
  • More
  • Click to share on Pinterest (Opens in new window)
  • Click to share on Tumblr (Opens in new window)
« Bring Your Own Exploit
The Whole Nine Yards with DevOps and Agile Operations »

Newsletter Sign-up

  • Notice: Submission of this form includes an automatic subscription to the DevOps e-newsletter. DevOps.com Privacy Policy

Upcoming Webinars

Thu 26

Mind the Gap: Bridging the Divide Between the Business and Agile/DevOps Teams with Value Stream Management

Thu, April 26, 1:00 pm - 2:00 pm EDT
Fri 27

Top 5 Takeaways from the 2018 Global Developer Survey

Fri, April 27, 11:00 am - 12:00 pm EDT
May 01

Enforcing Immutability and Least Privilege to Secure Containerized Applications on OpenShift

Tue, May 1, 11:00 am - 12:00 pm EDT

More Webinars

Past Webinars

Download Free eBook

Minimizing the Risks of OpenStack Adoption
Minimizing the Risks of OpenStack Adoption

RSS DevOps Chat

  • DOES London 2018 Preview w/ Mirco Herring, DevOps for the Modern Enterprise
  • Anything You Can Do With DevOps, You Can Do In A Mainframe, Chris O'Malley, Compuware
  • Low-Code Is Coming To An App Near You, Appian World Preview
  • Microservices on Google Cloud Update with Tim Hockin
  • DevOps Chat: Interop China Features DevOps & DevSecOps

Past Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays
  • Facebook
  • Google+
  • Linkedin
  • Twitter
  • Home
  • Business Directory
  • About DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

© 2018 · Mediaops, LLC.