The entire world is trying to move to Kubernetes. At the same time, everyone is afraid that they are going to do something wrong. They fear shipping apps and services that are poorly configured, wildly over-provisioned, or wildly over-permissioned. That fear is understandable, but it doesn’t need to be that way. You can adopt Kubernetes more securely by putting plans in place to help you prepare for and prevent issues related to misconfigurations, provisioning, and permissioning — in other words, you can start with the three Kubernetes security strategies discussed in this post.
Every organization or security team dreads the possibility of an unexpected impact to your applications, your data, your interaction with customers, your reputation. Whether that is a breach, applications being able to cross talk to each other in a way that you did not expect, or an expanded attack vector via a third party, if an attacker breaches a perimeter, they will have access to systems and services and may be able to cause considerable damage.
The best approach to Kubernetes security is to have your plan set up early on, because security is extremely hard to do retroactively. If your company is already five years old, your applications and services are built. You may have a complicated tech stack at that point, and you are likely to have significant technical debt built up. It will be much harder to implement the kind of security controls you need than if you had done it up front.
One substantial difference when you move to Kubernetes is that you are giving more control to individual development teams to provision infrastructure for themselves within Kubernetes. And with that extra power comes responsibility. Suddenly, those developers can do all sorts of things that they were not able to do before; in the past, they had to go through a DevOps engineer who spun up an EC2 instance for them. In Kubernetes — depending on the configuration, they can:
Run a command and have an application running inside the cluster
Run an application as root
Access files on the host node
Access secrets in the cluster
If you are trying to secure and manage your Kubernetes environment, you might want to prevent them from doing those things. That extra power individual development teams have makes Kubernetes a vastly different security profile than more traditional infrastructure platforms. Another thing to remember is that while Kubernetes is ephemeral, that does not protect your apps and services from security risks.
There are three elemental things your Kubernetes security strategy needs to cover:
Do not allow people to do more than they should
Do not allow avoidable known errors
Segment your people and teams by role
Let’s explore each of these topics in more depth.
Start with the principle of least privilege — limit access rights to those that are required for users to do their jobs. It’s not just for people, either. Least privilege access can apply to applications, systems, and processes as well. Below are examples of how that applies in Kubernetes.
When you are running a Docker container, you can pass around configuration information for it that says what permissions it has relative to the machine it is running on. One of the things you can do is add capabilities to a container to provide more access to the host machine. Unless your application is handling networking or baseline utilities as part of your infrastructure, you do not need to allow those insecure capabilities. Your average application does not need any special permissions or capabilities. Kubernetes does enable you to add those capabilities, however, so make sure that most teams under most circumstances are not adding those capabilities.
Privilege escalation means that the container can escalate to have more privileges. In the security context of your container, you must set allowPrivilegeEscalation=false. That helps you mitigate the threat if someone escapes a container via an exploit. Setting it to false means that an attacker will not be able to escalate to having root or administrative access on the host where your container runs. That container is running on a Kubernetes node where other containers and even different applications with diverse types of data are running, so you want to make sure that it only has the privileges needed and cannot escalate them.
This is a combination of insecure capabilities and privilege escalation. Disable the ability to run as a root user so that a potential container escape has fewer permissions and can do less damage. If you give extra permissions to a container, it can affect the host node and potentially other containers on that node. Disabling that capability adds an extra layer of protection to prevent containers from running as root. You might need to do a little bit of rearchitecting within your application to make that happen. This is a best practice for Kubernetes security.
This may sound like an easy security strategy to follow, but it can be difficult to avoid known errors as you are starting out. An experienced Kubernetes engineer can help you avoid all of the node errors or other mistakes you may make, but there are also a lot of open source tools and other third party tools available. These exist because a Kubernetes engineer sat down and wrote software to help keep people from making those top ten mistakes. Consider looking at the software or bringing in a Kubernetes expert, because it will save you considerable time and stress overall.
Every container you run in your Kubernetes cluster starts with a base image. That base image could be a vanilla operating system like Ubuntu or Alpine, or it could come with a bunch of extra software, like Python or NodeJS.. Then you install other things on top of that. You need to make sure that that software in your base image, as well as the things you install on top of it, are up to date and do not have vulnerabilities. Vulnerabilities can happen at the time you deploy an app, but they can also be discovered or disclosed later. You need to constantly assess your cluster for images that have vulnerabilities. Then you need to address those vulnerabilities and rebuild and redeploy your apps and services so the container image it lives inside of does not contain those vulnerabilities.
There are new vulnerabilities disclosed every day. Sometimes these vulnerabilities exist in code that has been running in your environment for a (very) long time. One way you can mitigate this risk is to minimize the number of unnecessary things you install in your container images. It may be tempting to put in tools to help you debug inside your container images (in case you need them someday) or other tools that are useful in a non-production environment. Then, when you promote that same image into production, you have an image that includes elements that are not necessary for your app to run — and increase the risk of image vulnerabilities in production environments. Instead, use commands such as kubectl debug to start a temporary ephemeral container alongside your workload for debugging purposes.
Kubernetes has a concept of role-based access control (RBAC). You can use role-based access control to grant privileges to the people who need to be interacting with Kubernetes. It is tempting to take the primary administrative role included with Kubernetes when you first set up a cluster and give that role to all the people in your small company because it helps you get moving quickly. Don’t do that.
Instead of an administrative-access free for all, use your own cluster roles and user roles to grant privileges using the minimum privilege model that's considered security best practice. Grant the least privileges necessary. Some container images require that they start up as root and then drop privileges, which means you must run them as root for a little while and then make sure privileges change. Make sure you work within the constraints of the roles, but don't turn the constraints off.
Audit logging is related to RBAC and can help you by providing a chronological set of the sequence of actions in a cluster. As you are building your cluster and operationalizing your team around Kubernetes, investigate audit logging, how you can review those logs, and how to tune them to minimize noise. The audit log shows you who is taking actions and how it relates to the roles and the access you enabled for each of them.
We collect statistics across organizations about what they are getting right and what they are getting wrong in their Kubernetes environments. Across organizations, we see that 54% of organizations are leaving over half their workloads open to privilege escalation, and thus there are security holes. It is worth looking at what you have implemented in your own cluster to make sure your workloads aren’t open to privilege escalation. Luckily, there are open source tools that help you put these Kubernetes security strategies in place.
Polaris comes with 30+ built-in checks based on Kubernetes best practices and supports custom policies. Using Polaris, you can write JSON schema to run against any resource in your Kubernetes cluster and create your own checks. Custom policies come into play when you get past broad best practices and start putting policies specific to your organization in place. Polaris can help you make sure that you do not allow people to do more than they should — the first Kubernetes security strategy outlined in this post.
Nova alerts you to available updates for helm charts deployed in your cluster. Updates are often related to a patched security vulnerability or newly available features, so Nova can help you increase your Kubernetes security posture. Pluto provides advanced warning of deprecated Kubernetes APIs. As Kubernetes gets updated with new features or improved implementation options, Kubernetes APIs change. Sometimes API deprecations and removals can be a surprise for Kubernetes users; Pluto helps you check your resources and your cluster regularly for deprecated APIs that may be removed in a future version of Kubernetes. Trivy has both an open scanner and an open database, and it can look for security issues and targets where it can find issues. Targets that Trivy can scan include container image, filesystem, Git repository (remote), Virtual Machine Image, Kubernetes, and AWS. These open source tools help you avoid known errors and keep your Kubernetes clusters humming.
For role-based access control, we also developed an open source tool called RBAC manager. It helps you simplify the amount of code you need to write to create a cluster role or role and bind that new role to users or groups of users.
It’s fairly straightforward to install open source projects on a single cluster and keep the projects up to date, however, it is more challenging to do that across tens or hundreds of clusters. In large Kubernetes environments, consider adopting governance software that helps you operationalize these Kubernetes security strategies from CI/CD through to production. Fairwinds Insights helps you increase visibility into security and compliance risks, optimize costs, and put guardrails in place for policies and security.
As 2022 comes to a close, how are you thinking about Kubernetes security in 2023? We know that DevOps teams are pivotal to securing Kubernetes, but are you ready for Kubernetes security strategies to evolve in 2023? These three strategies will put you on firm footing for the new year, and far closer to DevSecOps than you’ve been in the past.
