Brad Geesaman has been at the intersection of IT security and infrastructure for nearly two decades. An expert equally comfortable talking at KubeCon or BlackHat, Brad applies his cybersecurity expertise and container orchestration know-how as the co-founder of Darkbit, helping clients secure their cloud-native workloads. An active contributor in the Kubernetes community - he’s collaborated with a large and diverse group of people from all aspects of the community - Brad recommends becoming an active member of the Kubernetes collective as a first step to bringing Kube products to market. It’s this community focus and listen-and-learn style that makes Brad a perfect first interview for the Kubernetes Clinic: Spotlight blog series.
I got the chance to ask Brad my most pressing questions about the state of security and Kubernetes. From the need to trust that Kubernetes is being developed with security in mind to organizations’ over-reliance on a few key Kubernetes pros to why the OSS community keeps security risks quite low, Brad lays out what makes him nervous and what he’s excited about. He even shares why he thinks Kubernetes managed services - like Fairwinds - is a smart place for many organizations to get started with Kube.
We’re delighted that Brad sat with us to answer our questions about Kubernetes, security and the type of values he brings to his work at Darkbit. To thank him for his time, Fairwinds has made a donation to Women in Security & Privacy (WISP, www.wisporg.com) in Darkbit’s name. WISP is an organization Darkbit has donated to in the past because the team wants, “to continue to support the career advancement and thought leadership of women in the information security industry.”
Let’s get started with more from Brad Geesaman:
We’re doing pretty well, considering. Thank you for asking! I hope you all are doing as well as you can right now, too.
In early 2016, about a year after we were acquired by Symantec, our team of about six engineers was looking to redesign our Ethical Hacking Simulation platform that we used to run Capture-the-Flag events from virtualization on bare metal to AWS. The primary goals were cost efficiency, faster target development, quicker target spin up, dedicated targets per team of 4, better geographic distribution, and automating everything as code. We had about 9 months before we were going to host 1500 world-wide participants for a 1 week, 24x7 online event. The natural thought was leveraging EC2 instances in different AWS regions, but that only solved some of those goals well. Our team was already leveraging Docker containers for some target development, and we decided to do a proof-of-concept on Docker Swarm and Kubernetes 1.3. Despite not having RBAC and Calico Network Policies being in alpha, Kubernetes had the flexibility to do exactly what we wanted. Purposefully allowing participants to get a shell within one or more containers in their own namespace was an early form of the multi-tenancy problem we see today. And we set to penetration testing our own platform and locking things down one by one with that in mind. That hands-on security education of the Kubernetes components has proven invaluable ever since.
It all comes down to trust. Trusting that the Kubernetes features are being developed with strong security in mind. Trusting the defaults of the cloud to be secure in terms of configuration. Trusting the source of container images, their dependencies, and the security of the people that contribute that code. There is a lot of assumed trust throughout the ecosystem that organizations need to really understand well to ensure they are operating at the desired levels of security. As a community, we need to be constantly revisiting these assumptions because users are relying on these decisions to deliver their products and services.
I see several things consistently: From an organizational perspective, an over-reliance on one or two key individuals that understand the entire system. They tend to be a bottleneck for those teams trying to scale and are exceedingly prone to burnout. From an operational perspective, we see symptoms of hitting a “complexity” wall. By that, I mean there is some point where the environment, the cluster, and the apps start working well enough and adding things like tighter security policies starts to break that success and require too much cognitive load. Running images from DockerHub without vetting, lack of pod network segmentation, and lack of a coherent strategy for detecting malicious behavior inside clusters are common side effects of that.
I see the next 1-2 years as a real inflection point for the Kubernetes project to balance the need for rapidly accepting features and contributions that some users need with the security and stability that many other users need. One option could potentially be a Long-Term Support (LTS) release approach that could give some organizations a more stable target and relieve that update pressure on their operations teams.
There are two ways to answer that question. One, for someone who’s started a company centered around Kubernetes products and services. Two, for organizations adopting Kubernetes and looking for security advice.
For the former, given the explosive growth of the project and the surrounding community projects, it may be tempting for newcomers to the community to see a perceived gap and try to immediately develop a product around that. I’d strongly recommend spending time becoming a contributing member of the community first and as much time as possible with actual Kubernetes users at all stages in their journey to really understand the underlying culture. Otherwise, it’s very likely that your products and/or services will suffer from technical and cultural mis-alignment.
For the latter, most use cases, leveraging the cloud provider’s managed offering is the best starting point. However, they do not provide the level of security desired by most organizations with their out of the box configuration. Spend some time upfront getting a handle on the security-related settings available, and build those into your infrastructure-as-code approach early and often. Then, focus the energy and resources on automating and securing your workloads.
Given the enormous amount of contributors, there are many eyes on proposed changes and code reviews, so the risk of submitting intentionally malicious code is quite low. However, when a project is this big with this many moving parts of ever-growing complexity, the huge challenge is ensuring that any new features or changes are understood from a security perspective across all components in all potential environments. Also, as behavior of a component becomes relied upon, a newly discovered security issue that requires a fundamental change to that behavior becomes much more difficult to make due to its inertia. It will be a continuous challenge to balance those decisions as the project continues to grow, for sure.
I think there is a balance of upstream release cadence and speed at which platform teams can consume them, and the pace of release consumption is slowing for reasons of increasing feature complexity or because that release is doing everything they need at the moment. It’s not a new concept that organizations de-prioritize security patching, but this is definitely exacerbated by the shorter release support lifetimes. At some point, there will be some level of pushback to help restore this balance. That said, I’m more concerned about the ability for organizations to manage their container image security as they wrestle with “container sprawl”.
I’m extremely humbled to have worked with so many outstanding folks in this community, and collaborating with Ian for RSA2020 and again for KubeConEU2020 has been a great experience! I listen whenever Rory McCune, Duffie Cooley, Tabby Sable, Liz Rice, Michael Hausenblas, Jordan Liggit, Andrew Martin, Maya Kaczorowski and so many others share their thoughts (or exploits!).
At Darkbit, our core values are trustworthiness, transparency, empathy, and learning. We strive to be open and honest about everything we do, to continuously improve, and we recognize that we are part of a much larger community all pulling in the same direction.