In our series on the NSA Kubernetes Hardening Guide, we’ve looked at pod security, network access and authentication and authorization. Today we look at the audit logging and threat detection section and offer some suggestions on compliance.
The NSA Kubernetes Hardening Guidelines outlines a strong defense-in-depth approach to minimize the chances of a breach, and to ensure that if an attacker does infiltrate your cluster, the blast radius will be as small as possible. Audit logging and threat detection are an important part of Kubernetes security.
Audit logs capture attributed activity in the cluster. An effective logging solution and log reviewing are necessary, not only for ensuring that services are operating and configured as intended, but also for ensuring the security of the system.
The Kubernetes Audit Logs, when fully enabled, generate a firehose of information. At a bare minimum, you should enable Audit Logs and ensure they're stored in a place where they can be reviewed in case of emergency. Audit Logs may help you get to the bottom of a security breach or an outage when other, more easily digestible telemetry fails.
Logging should be performed at all levels of the environment, including on the host, application, container, container engine, image registry, api-server, and the cloud, as applicable. Once captured, these logs should all be aggregated to a single service to provide security auditors, network defenders and incident responders a full view of the actions taken throughout the environment.
We recommend sending all audit, node and application logs, as well as any other relevant logs, to a third-party service like Datadog. This move helps to ensure the logs persist in the event of an attack or outage. It also serves to collate logs from disparate sources, so incidents can be investigated through a single pane of glass.
Fairwinds Insights, a Kubernetes governance platform, can export events and metrics to Datadog, allowing you to annotate graphs and enhance logs with additional information related to changes in your Kubernetes environment.
System administrators running applications within Kubernetes should establish an effective logging and monitoring system for their environment. Logging Kubernetes events alone is not enough to provide a full picture of the actions occurring on the system. Logging should be performed at all levels of the environment, including on the host, application, container, container engine, image registry, api-server and the cloud, as applicable.
Ensuring you have full visibility into every level of the stack is critical for both the security and reliability of your clusters. This can be difficult to configure manually.
We recommend utilizing third-party software like Datadog to aggregate logs and persist them to an external environment. The Datadog agent automatically ingests node and application logs from every workload in your cluster by running a DaemonSet (which places a Pod on each node in the cluster).
The NSA also recommends some lower-level logging utilizing seccomp in audit mode to log system calls on the host nodes. We recommend using Falco, a cloud-native runtime security open source project, which adds the additional benefit of matching particular sets of system calls (including network traffic patterns) to flag suspicious behavior. Falco comes with a large set of built-in patterns, and additional patterns can be defined by the organization.
Fairwinds Insights can be used to ingest Falco findings and send alerts when suspicious events occur.
Kubernetes does not natively support alerting; however, several monitoring tools with alerting capabilities are compatible with Kubernetes. If Kubernetes administrators choose to configure an alerting tool to work within a Kubernetes environment, administrators can use several metrics to monitor and configure alerts.
Logging alone can be great for auditing purposes, but its usefulness is limited if you don't have monitoring and alerting set up.
Some organizations rely on open source solutions here, usually a combination of Prometheus and Grafana. While this solution is good enough for some cases, self-hosting your monitoring and alerting is difficult and prone to cascading failures.
We recommend using a combination of Datadog and PagerDuty to configure monitoring and alerting. Datadog can create fine-tuned alerts based on both raw metrics and log contents, and can send alerts out to PagerDuty, Slack or any other place your engineers might want to get alerted. PagerDuty can help you manage pager rotations among your engineers, so you can avoid single-points-of-failure and prevent burnout.
Fairwinds Insights also comes with built-in alerting and connects natively to both Datadog and Pagerduty, so you can alert on any security, reliability or efficiency issues introduced into your environment.
Fairwinds Insights, a platform for Kubernetes governance and security, can help accomplish many of the NSA’s most important guidelines. Utilizing Fairwinds Insights, in conjunction with other best-of-breed commercial and open source software, can help organizations’ achieve compliance with the NSA’s recommendations. Fairwinds Insights is available to use for free. You can sign up here.
