DevOpsQA NJ Meetup - ELK Stack

Our guest speaker at the May DevOpsQA NJ Meetup was Peter Kim from Elastic. Peter is a Solutions Architect at Elastic with over 10 years of experience architecting and developing search applications with Endeca, MarkLogic and core Lucene. He gave us a great overview of ELK Stack, which consists of ElasticSearch, LogStash and Kibana.

In summary, ELK Stack is an end-to-end stack that delivers real-time insights into data from almost any type of structured and unstructured data sources. It was designed to address the issues of inconsistent log formats, log distribution across servers as well as high level of expertise required to understand the log outputs. ELK Stack provides open source integration with configuration management tools like Chef, Puppet and Ansible as well as a connector for Hadoop.

Peter then extensively discussed each component of ELK stack. Logstash manages events logs, by collecting, parsing and storing data. It is fully open source with the Apache 2.0 license . Grok is a tool that can be used to parse unstructured data into something structured and queryable.

ElasticSearch is schema-free, REST & JSON based document store. It’s every function is exposed via REST APIs. ElasticSearch is distributed and horizontally scalable, comes with a log of plugins for analysis, localization support, etc and client API for various programming languages.

Kibana is a data visualization and analytics platform integrated with ElasticSearch . Peter gave a live demo of Kibana with Apache weblog data.

The session was followed by a lot of good questions ranging from security, real-time data to the fuzzy search and complex aggregations. At the end, Peter told us about Elastic Roadmap including Watcher for alerting and new features which are coming out as part of ElasticSearch 2.0.