In our second blog covering the NSA Kubernetes Hardening Guidelines, we will discuss the myriad ways Fairwinds Insights—along with open source and cloud native technologies—can help your organization comply with these important recommendations. While our first piece covered the capabilities of Insights to enhance and bolster pod security, something all businesses in the cloud native landscape should be concerned about, our second dive will focus on network separation and hardening.
Want all the NSA recommendations at once? Download our newest white paper,
Steps to Meeting NSA Kubernetes Hardening Guidelines.
Cluster networking is central to the concept of Kubernetes. Communication among pods, containers and external services must be taken into account. Kubernetes resources, by default, are not isolated and can’t prevent lateral movement or escalation if a cluster is compromised. Resource separation and encryption can be effective methods for limiting a digital attacker’s movement and escalation within a Kubernetes cluster.
Big picture, the NSA recommends using network policies and firewalls to separate and isolate resources, secure control planes, as well as encrypting traffic and sensitive data at rest. Let’s look at the different ways our Kubernetes governance software addresses these critical areas, including best methods for complying with all measures.
What are the NSA recommendations around network separation?
NSA: Lock down access to control plane nodes using a firewall. The control plane is the core of Kubernetes and allows users to view containers, schedule new pods, read Secrets and execute commands in the cluster. Because of these sensitive capabilities, the control plane should be highly protected. The Kubernetes API server should not be exposed to the internet or an untrusted network.
The easiest way to secure access to the control plane is to use a managed Kubernetes service like GKE or EKS. These solutions abstract away the control plane, protecting it from misconfiguration.
If you do have to manage your own control plane nodes, use things like VPCs and Network Access Control Lists (NACL) to control access to the nodes. We recommend gating access to the control plane through an SSH bastion in the same VPC as your control plane nodes.
NSA: Lock down access to control plane nodes using RBAC. Enabled by default, RBAC is one method used to control access to a cluster resources based on the roles of individuals within an organization. RBAC can be used to restrict access to user and service accounts.
RBAC is a tricky concept in Kubernetes. Fairwinds provides two open source projects for helping to manage RBAC:
While every organization will have different needs and make different design decisions about how to provision RBAC, it’s imperative to make use of Roles and ClusterRoles to provision different levels of access for different personas. This helps with adherence to the principle of least privilege, with the added benefit of preventing honest mistakes.
Fairwinds Insights offers an interface for auditing RBAC within every cluster, surfacing particularly dangerous Roles and ClusterRoles.
NSA: Use separate networks for the control plane components and nodes. An administrator should proactively limit the attack surface by separating the worker nodes from other network segments that do not need to communicate with the worker nodes or Kubernetes services.
Network segmentation is a great way to limit the blast radius of an attack. In particular, it’s important to keep user-facing applications separated from the control plane, so an attacker is prevented from escaping the application environment and gaining access to the entire cluster.
We recommend using different subnets within your VPC for different types of worker nodes, specifically separating out tooling like nginx-ingress or cert- manager, which need control plane access, from user-facing applications. Concepts like AWS’s Security Groups can then be used to restrict communication between node groups.
With Fairwinds Insights, you can run kube-hunter inside your Kubernetes cluster to better understand which nodes have control plane access. To get a sense of what access those nodes have to the control plane, we recommend configuring kube-hunter to run on application nodes.
NSA: Configure control plane components to use authenticated, encrypted communications using TLS. The etcd backend database stores state information and cluster Secrets. It is a criticalcontrol plane component, and gaining write access to etcd could give a cyber actor root access to the entire cluster.
The majority of Kubernetes deployments will encrypt network traffic by default. Whether you’re using a managed Kubernetes service like EKS or GKE, or a tool like kOps to manage your own control plane, the default configuration will use TLS to encrypt communication with the control plane.
As always, make sure you’re using up-to-date versions of Kubernetes and any tooling like kOps.
NSA: Encrypt etcd at rest and use a separate TLS certificate. The etcd backend database is a critical control plane component and the most important piece to secure within the control plane.
Again, by using a managed Kubernetes service like GKE or EKS, you can avoid having to worry about your control plane configuration. But if you do need to manage your control plane nodes, be sure to encrypt the underlying volumes, as well as set up a unique TLS certificate for communication with etcd.
Fairwinds Insights can run kube-bench in each of your clusters, which runs many automated checks against the CIS benchmark. One of these checks (check 2.1) will ensure that you’ve configured etcd with an appropriate -cert-file and --key- file. If not, Fairwinds Insights will create an Action Item to fix the issue.
NSA: Set up network policies to isolate resources. Network policies control traffic flow between pods, namespaces and external IP addresses. By default, no network policies are applied to pods or namespaces, resulting in unrestricted ingress and egress traffic within the pod network.
While Kubernetes comes with a built-in NetworkPolicy resource, it is fairly limited in scope. It can only restrict traffic at L3/L4, e.g. based on IP address, which makes it very hard to use in practice.
We recommend using an open source service mesh like Linkerd. This service can capture L7 semantics, allowing for more readable, maintainable and fine-grained network policies. Linkerd also enforces its policies at the Pod level rather than the CNI level, which gets closer to the “zero trust” ideal.
If you are interested in hearing more about how Insights can help your organization achieve other benchmarks from the new NSA guidelines, including how to create an explicit deny network policy and place all sensitive information encrypted in Kubernetes Secrets, read our new white paper, Steps to Meeting NSA Kubernetes Hardening Guidelines. Furthermore, this information will inform you on how to avoid misconfigurations and implement recommended measures and mitigations during deployment.
Our NSA white paper also provides information on meeting all the NSA recommendations, not just pod security and networking hardening. It includes detailed discussion on other areas like authentication and authorization, audit logging, threat detection and other application security practices. As the containerized world continues to evolve, it becomes increasingly clear that a strong defense-in-depth approach is the best way to minimize risk and maximize efficiency and innovation.