IBM Open Sources SysFlow Monitoring Platform

IBM has announced that its SysFlow monitoring platform is now available as an open source project.

Fred Araujo, a research scientist in the Cognitive Cybersecurity Intelligence Group at IBM Research, said IBM developed lightweight SysFlow agent software and monitoring tools as a way to provide more context around the telemetry data being collected while simultaneously reducing the amount of data that needs to be stored.

SysFlow encodes a representation of system activities into a compact format that records how applications interact with their environment, Araujo said, noting that level of context provides deeper visibility in everything from container workloads to cybersecurity forensics. However, unlike existing monitoring platforms, SysFlow doesn’t require IT organizations to collect a massive amount of data to achieve that goal—it is intended to provide for a superset of the NetFlow framework used to analyze network traffic patterns to capture system events, he said.

Araujo noted IBM doesn’t envision SysFlow eliminating the need for legacy log analytics platforms, as they provide a way to analyze log data. However, SysFlow does enable IT organizations to apply analytics via a graph-like visualization to surface patterns that goes beyond a comparative simple rules-based approach, said Araujo. For example, SysFlow’s approach will make it easier to uncover the relationship between various events that make up a cybersecurity attack and subsequently to identify what countermeasures to employ to create the appropriate kill chain response. It also should substantially reduce the amount of fatigue cybersecurity teams experience from chasing down false-positive alerts, he said.

SysFlow is designed from the ground up to integrate with both open source frameworks such as Apache Spark and commercial analytics platforms via an open serialization format and associated libraries. IT organizations also can leverage a set of reusable components and APIs to make it easier to deploy telemetry probes. IT organizations also can take advantage of an extensible policy engine that can ingest customizable security policies described in a declarative input language, which then can be checked against records captured by SysFlow.

Araujo said IBM developed SysFlow to address the challenges associated with managing application workloads in public clouds, but the framework can be applied to any on-premises IT environment to create a single pane of glass through which multiple platforms can be monitored.

Regardless of the platform on which a workload is deployed, Araujo said as more responsibility for managing and securing applications shifts left toward developers in the age of DevOps, those teams need access to more sophisticated tools that are readily available and simple to deploy. There are several open source initiatives underway to provide that visibility, and one day many of them might coalesce into a single initiative.

In the meantime, as the dependencies between applications and IT infrastructure continue to increase, it’s clear IT monitoring as a core part of any set of best DevOps practices is now an absolute requirement to achieve and maintain observability.

— Mike Vizard