Securing workloads in Kubernetes is an important part of your cluster security overall. Your overall goal should be to ensure that your containers are running with minimal privileges (as much as possible). Configuring Kubernetes workloads for ease of use, unfortunately, can create time consuming and costly security vulnerabilities. To minimize the risk of misconfigurations and vulnerabilities, you must work to avoid privilege escalation, ensure that you are not running containers as a root user, and are using read only file systems wherever possible.
It’s easy for security misconfigurations and vulnerabilities to slip into production environments in Kubernetes because of lack of oversight or experience. Speed to delivery vs. critical security safeguards are often at odds as teams attempt to balance the velocity of engineering with the often less rapid pace of security. This balancing act can result in messy Kubernetes configurations and unnecessary risk. Problems can arise if workloads are misconfigured by developers through inexperience or lack of awareness of the issue. And, sometimes, organizations assume that cloud service providers handle all aspects of cloud security, but that simply isn’t the case.
While cloud providers, such as Microsoft Azure, Amazon AWS, and Google Cloud Platform (GCP), handle some aspects of cloud security, it’s a shared responsibility model. Cloud providers are responsible for the security ”of” the cloud, including physical facilities, utilities, hardware, cables, and so on. Customers — those deploying applications and services in cloud environments, are responsible for security “in” the cloud — that means securing sensitive data, applying network controls, handling identity and access management (IAM), application configurations, and more. You must put security controls in place to protect your web applications and application stack and ensure that authentication and authorization rules are being followed. Data breaches are often traced back to cloud misconfigurations, which is why it’s important to be aware of how to properly configure and secure containers and Kubernetes effectively.
Our recent Kubernetes Configuration Benchmark Report tells us that not all organizations have found steady footing with proper configuration of their Kubernetes environment. The report shows that Kubernetes security, cost efficiency, and reliability remain top concerns to cloud native users. According to the benchmark data, teams continue to struggle to configure Kubernetes so that it’s aligned to best practices. That lack of alignment translates into unnecessary risk, cloud cost overruns, and sometimes even lost customers due to poor application performance or data breaches.
Increasingly, federal agencies, such as the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA), are offering guidance on how to harden Kubernetes to reduce the risk of hackers and cyber attackers gaining access to these environments. The Federal Risk and Authorization Management Program (FedRAMP) also provides guidance on vulnerability scanning requirements for containers to enable a standardized approach to security assessments, authorization, and continuous monitoring that is focused on cloud products and services. The responsibility, then, is for the organization to follow that guidance using tools that help to meet requirements to scan container images for vulnerabilities.
The goal of these security frameworks is to minimize human error and provide guidance on how to prevent misconfigurations and remediate vulnerabilities quickly. Bad actors frequently leverage common misconfigurations, default accounts, default passwords, and human error to carry out ransomware attacks and other malicious activities. These frameworks also provide guidance on adopting the principle of least privilege and robust access control tools to minimize unauthorized access that could result in security issues in applications and services hosted in public cloud environments.
Kubernetes security misconfigurations frequently occur when configurations simply have not been set. Often this is because the fastest way to production is to move past the set up stage, prioritizing functionality over security. Minor misconfigurations can produce major security holes if they are not found and addressed, posing significant concerns for cybersecurity teams. Because configurations are not established by default, security-conscious teams need to explicitly set them. For example, a common vulnerability is containers running with more security permissions than required, such as enabling root-level access. Under particular configurations, a container may escalate its own privileges.
Unfortunately, individual application developers sometimes neglect or are unaware of security configuration needs for each workload. For example, it's often easier to over-permission a deployment with root access to just get something working. Forcing individual contributors to define and apply their own security configurations all but ensures inconsistency and mistakes.
Some sample security configurations in Kubernetes include:
Host IPC should not be configured
Privilege escalation should not be allowed
Container should not have insecure capabilities (like NET_ADMIN)
Container should not run as root
See this full list of security configurations. Some misconfigurations can be resolved automatically using Fairwinds Insights’ Automated Pull Request capability. This feature makes it easier for devs to make misconfiguration fixes quickly and easily.
Avoiding misconfigurations requires a few things. First, DevOps teams need to regularly scan Kubernetes clusters to ensure they have been configured correctly. Often organizations do this manually, with a spreadsheet, which is handed down to developers to fix. Unfortunately, those changes are not always made, so the cycle keeps repeating. The number one tip for avoiding misconfigurations is to scan clusters automatically and continuously from dev through to production. Doing so helps to avoid these misconfigurations from entering any production environment.
Better yet, using an Admission Controller to prevent misconfigurations can help to enable developers to configure Kubernetes better. The Admission Controller can be part of the CI/CD process, scanning containers for misconfigurations before they are checked in and pushed to production. The developer can be alerted of any issues before it goes into production. This helps enable developers to fix their own problems and enables DevOps to focus on infrastructure improvements rather than acting as a Kubernetes helpdesk for developers.
So how do you quickly and proactively identify Kubernetes security misconfigurations to prevent security risks? In our experience helping organizations with Kubernetes management, we see eight common Kubernetes security misconfigurations that lead to vulnerable deployments.
Not identifying and addressing these security settings and configurations can have negative business consequences. For example, if a container runs as root but doesn’t necessarily need this level of access, then a malicious container may have the privileges to steal data or cause other damage to the system.
|
Configuration |
Severity |
Description |
|
security.hostIPCSet |
danger |
Fails when hostIPC attribute is configured. |
|
security.hostPIDSet |
danger |
Fails when hostPID attribute is configured. |
|
security.notReadOnlyRootFilesystem |
warning |
Fails when securityContext.readOnlyRootFilesystem is not true. |
|
security.privilegeEscalationAllowed |
danger |
Fails when securityContext.allowPrivilegeEscalation is true. |
|
security.runAsRootAllowed |
danger |
Fails when securityContext.runAsNonRoot is not true. |
|
security.runAsPrivileged |
danger |
Fails when securityContext.privileged is true. |
|
security.insecureCapabilities |
warning |
Fails when securityContext.capabilities includes one of the capabilities listed here |
|
security.dangerousCapabilities |
danger |
Fails when securityContext.capabilities includes one of the capabilities listed here |
To address these common Kubernetes security threats, our team built Polaris, an open source tool, that checks configurations found in the securityContext attribute for both Kubernetes pods and containers. Where Polaris ends is where Fairwinds Insights picks up.
Fairwinds Insights is a configuration validation tool that provides visibility into an organization’s Kubernetes security posture by auditing workloads and validating configurations for weaknesses, container vulnerabilities, and misconfigured deployments. It can be easy for configuration drift to occur because there are often manual changes and updates on individual clusters, resulting in an infrastructure that becomes increasingly different over time. That makes it hard to identify and correct inconsistencies, which can result in security vulnerabilities and issues with efficiency and reliability.
Fairwinds Insights operationalizes Polaris checks by providing not only the findings, but also keeping a historical record of the results across all your clusters and offering remediation guidance. Fairwinds Insights allows you to prioritize and track security, efficiency, and reliability issues. It also enables collaboration across teams, enforces consistency in configurations, and enables you to apply best practices automatically as applications move from development to production.
Learn more about Fairwinds Insights or read this white paper on how to manage Kubernetes configurations to improve security, efficiency and reliability.
Originally published 11 June 2020. Updated and revised 11 May 2023.
