When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.
The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.
K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?
One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.
In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:
There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.
Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.
In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.
As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.
Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.
Datadog today published a State of DevSecOps report that finds 90% of Java services running in a production environment are…
Linux dodged a bullet. If the XZ exploit had gone undiscovered for only a few more weeks, millions of Linux…
We're going to send email messages that say, "Hope this finds you in a well" and see if anybody notices.
I am happy and proud to announce with Daniel Newman, CEO of Futurum Group, an agreement under which Futurum has…
Most developers are using some form of DevOps practices, reports the CDF survey. Adopting STANDARD DevOps practices? Not so much.
Two thirds of developers are using AI in product development, primarily for coding, documentation, and conducting research.