Talking to Tabitha Sable is like trying to stuff 20 pounds of goodness into a 5-pound bag – the whole thing overflows with insight, technical knowledge and a healthy dose of community support. Whether she’s reading code and documentation to learn what’s underneath, chewing on Kubernetes security or discussing why corporations need to see hiring a senior engineer as a strategic decision, Tabitha is smart, equally interested and interesting, and extremely giving of her time and knowledge.
We had the chance to visit with Tabitha recently, covering a wide range of topics, both related to DevOps security and not. There’s a reason her friends say they listen when she talks – she approaches her work and her life with purposeful thoughtfulness and is committed to collaboration inside our industry.
Read on for a charming, clever conversation that is equal parts technical and personal. In exchange for her participation in this Q&A, Fairwinds has made a donation to “Women in Security and Privacy” in Tabitha’s name. She chose this charity because she’s, “seen firsthand how WISP has connected people to opportunities that improve careers, change lives, and improve the industry.”
First, how are you doing in the midst of a pandemic and a complete shift in how we live and work?
I've been tremendously lucky through the pandemic so far. I miss seeing people, though! I've been mostly trying to focus on maintaining my emotional and physical health and using my privilege to support my communities. We gotta get through this together.
Our first “Kubernetes Clinic Spotlight” featured Brad Geesaman from DarkBit. Thank you for retweeting it. He said he listens whenever you have something to say. What do you think resonates so much between you and your audience?
My educational background is math and physics, and those disciplines taught me a lot about the value of showing your work and embracing being wrong about things. When I share my opinion about something, especially something technical, it's important to me to include the reasons why so that people can put their own context around it. I think people appreciate hearing the "why", and the honesty of that, in an industry full of "rockstars" and thought-leader "Uncle"s who argue from their authority or reputation.
I saw you wrote something recently that referenced the “emergent behavior of complex systems.” What do you see on the horizon regarding this topic as it relates to your work and our collective industry? What should we be thinking about?
As our tech stacks get more complex, that naturally encourages us to specialize. Specialization helps to maintain a deep understanding of our individual domains, but I think it also leaves us vulnerable to overlooking interactions that are hidden from us by our points of view. To ensure we cover each other’s weaknesses, we need to have generalists working together with specialists, breakers with builders, dev with ops. We have to stay curious and outgoing, because none of us can understand it all. I really enjoyed Ian Coldwater's KubeCon NA 2019 keynote "Hello From the Other Side: Dispatches From a Kubernetes Attacker", which touches on these ideas specifically around Kubernetes security.
I also think we need to be more intentional about optimizing the whole system. To properly balance features, security, and maintainability in a complex stack like an app running on Kubernetes, we must look beyond optimizing each component separately. Cross-functional collaboration helps form this kind of holistic vision. To understand these concerns better, I'm working through Westerman's "Systems Engineering Principles and Practice" right now. I'm always on the lookout for new reading material; feel free to ping me on Twitter with your favorite resources about systems engineering!
Your GitHub profile says, “gotta break things to understand what’s inside.” You’re a hacker who’s focused on containers and Kubernetes. How have you broken Kubernetes and what did you find inside?
Most of my exploration of Kubernetes has come via talking to people and reading the code. I love how frequently security research begins by reading code or documentation really, really closely. Sometimes you find important differences between the spec and the implementation, or creative uses of the specification that had been previously overlooked!
The key thing I've found inside Kubernetes is complexity. Because it’s huge and has so many features, I'm always astounded by how reliable and secure Kubernetes is. That security and reliability shows how deeply the Kubernetes community cares about quality. Despite that care, many security issues arise from Kubernetes's complexity. CVE-2020-8558 is a great example: the "route_localnet" sysctls change the behavior of Linux in ways that are quite foreign to most of us. The details of Layer 2 networking are pretty obscure and far removed from the day-to-day experience of cloud-focused engineers, so it's no surprise that numerous talented engineers overlooked how setting those sysctls could cause a security issue.
You co-created and co-presented “Attacking and Defending Kubernetes Clusters: A Guided Tour Walkthrough Guide” at KubeCon NA 2019, then made it available as a self-hosted session earlier this year. What sort of engagement have you gotten from your audience and were there any interesting reactions?
The energy in the room that day was amazing. All the seats were full, people crowded around the aisles or sat on the floor, and everybody was excited to be there. About a dozen people from the Kubernetes security community showed up unbidden to help support the audience. Attendees worked together, helping out their neighbors when they'd get stuck. There was a tangible feeling of shared joy when over 400 people said they'd just escaped a container for the first time.
Since KubeCon, my favorite reaction has been when people say they're adding our workshop to their onboarding programs. It's such an honor to help people level themselves up!
Kelsey Hightower said in his Container Security Summit session, “they say that security isn’t a destination, it’s a practice. But ain’t nobody practicing.” We’d love to hear your thoughts on this.
It was a delight to hear Kelsey's summary of the frustration that I share with so many of my DevOps security colleagues. Effective security has always been a practice, a state of constant learning and revisiting our past decisions when new information becomes available. But we have so many cultural problems in infosec that make it hard for us to live that life together! We look down on our teammates in dev and ops. We treat compliance requirements like authoritative wisdom rather than problematic suggestions. Our orgs try to "sprinkle security on top" just before a project ships. We have a whole entrenched computer security industry based on an us-vs-them idea that security needs to save the org from everybody else's supposed incompetence.
That attitude never really gave great results, but in the on-prem '90s the candybar network (crunchy on the outside, chewy on the inside) was often good enough. Today, with your cloud provider IAM configuration taking the place of the locks that used to be on your computer room door, infosec can no longer afford to be so dysfunctional. We have to build relationships so we can be invited to the room where systems are designed. We need to understand our dev and ops colleagues' needs well enough to suggest realistic solutions to their security challenges. "The department of saying no" has gotta go.
You’ve been on the speaking circuit around containers, Kubernetes and security. Has anything really stood out to you in the past 6 - 12 months that we need to pay closer attention to or learn more about?
There's lots of cool advanced technology that I'm excited about, but most of us have a lot of room to level up our operational capabilities. Security basics are still really challenging. Orgs frequently struggle with knowing what's running, quickly and easily deploying patches or config changes, and collecting the right signals to detect security incidents. Cloud native infrastructure gives us tools to pin down these foundational practices better than ever before, but it's not automatic. Succeeding requires intentional, thoughtful collaboration and a holistic shared vision of what "doing ops really well" means. Going back to the previous remark about "ain't nobody practicing", security and compliance teams have important contributions to make here also. They're in a great position to provide monitoring and close feedback loops. It's not about security pointing fingers, it's about helping ops do their best. The result is better security and also lower costs, better reliability, and higher velocity. Everybody wins.
We saw a Tweet of yours from earlier this year regarding different types of security communities, from product to infrastructure to offensive or defensive security. How do we help a product security professional think like an offensive security pro, etc.? And why is it important?
As security practitioners, I think we can align more by focusing on our shared responsibility to help users be safer. It's both the end and the means. Centering our conversations around the user helps us make our work more practical, and it also gives us a shared place to start cross-team conversations.
Talking with others outside our specialty is the best way I know of to build understanding of how our work fits together. Seeing each other as individuals with different day-to-day concerns but the same end goal helps reduce the finger-pointing that so often undermines our efforts to improve security. Basically, make friends and fight together for your users. It's more fun than going it alone, and it works better too.
Your Twitter feed is full of /honks and praise for your industry colleagues and you collaborate on talks with a wide number of them. Can you tell us about your approach to collaboration?
I love that Fred Rogers "look for the helpers" quote, and honestly that's my approach to collaboration. As a technologist, I have plenty of forces in my life that encourage me to spend time alone, so I try to actively counter that by seeking out collaborative work. Sometimes that means getting an exciting idea and rushing to DM it to a colleague with a shared interest; other times, it means yelling "yes!" when somebody else shares their squee.
It can feel mercenary to purposefully build your professional network, but all that really means is "meet other people doing cool things and try to remember which ones have the kind of attitude you want to be around." When you work on a project with someone, you spend a lot of time together and influence each other. Therefore, it's important to me to focus on group projects with people who lift up and encourage those around them, because lifting as I climb is my most fundamental goal. It forms a positive feedback loop, and that's how we can change the culture of the industry at large: one person at a time, by caring about each other.
You recently did a really wonderful interview about your career path that resonated with a lot of people, including me. One thing that caught my eye and that I wanted to ask about was that hiring an engineer at a very high level is a strategic decision for companies. Can you tell us more about that both from the position of the engineer and the company?
A very senior engineer doing their best work transforms the organization around them. By choosing to hire a very senior engineer, an org is choosing how it wants to be transformed. This is an important strategic decision. When debriefing after a staff engineer candidate interview panel, I like to ask the question "What big changes can we make by hiring this person, and are those the changes we want to make?"
One of the ways a very senior engineer transforms the organization is by guiding major engineering decision-making processes. A beginning engineer's first task is to learn one way of solving a problem. An important sign of growing mastery is to envision many alternatives and the consequences of each, including the trade-offs between quality, timeline, cost, maintainability, performance, and so on. A staff- or principal-level engineer needs to do this while considering the needs of the organization holistically, including both short- and long-term concerns from business and engineering points of view. This is part art and part science, and the judgment applied to these matters by a very senior engineer has long-lasting effects across the organization.
Additionally, a very senior engineer is a role model with a powerful effect on the engineering team culture. Their position on the org chart is an explicit endorsement by the organization. Others will seek to emulate that engineer's skills, habits, practices, and attitude. A very senior engineer will naturally be involved in far more tasks than any individual can execute on by themselves. Do they build learning and growth by encouraging other engineers to take on tasks that are appropriately challenging? Do they provide guidance and assistance to ensure those difficult tasks succeed? Are they fair and kind (which is different than nice!) in their interactions with other engineers? The answers to these questions are key inputs that drive the engineering team's cultural evolution.
For the engineer, I think it's important to keep in mind that your success as a staff- or principal-level engineer hinges on having an opinionated vision for technical advancement. That vision is what can transform the organization. You need to be able to work across all levels of the org, from senior leadership to junior individual contributors, to make that vision become reality. You need to do so without being dogmatic; you have to recognize when to defer to others or meld their ideas with your own to achieve a better outcome than any of you could have made alone. In my opinion, that balance is what you're selling during a very senior engineering job search.
In that same interview, you talked about “low-key sexism” that comes into play in tech interviewing. I thought your commentary here – really the whole interview – was extremely thoughtful and important. Are you seeing progress re: sexism in interviewing or Diversity, Equity & Inclusion (DEI) overall or are we still struggling at a baseline?
I think tech is making slow progress in all aspects of DEI, and the progress is unevenly distributed. Some orgs are doing great while others are struggling. Within an org, often some teams are doing great while others are struggling. The ones doing better usually have diverse leadership.
Winning organizations are working hard to gain the advantage of being able to hire and retain employees from the whole workforce. They look for input from their marginalized team members, reward the effort that providing that input requires, and are respectful that not every member of an under-represented minority wants to spend their time on DEI work. Those successful orgs also look beyond their team, with leaders reading and learning from blog posts and essays, conference talks, seminars, and so forth.
The demographics of tech as a whole are still extremely lopsided, so an org can struggle with DEI simply by default. If everyone is from basically the same background, it can be easy to overlook aspects of your culture that drive away people who aren't like you. The result of that carelessness is that you don't have the quality of talent you otherwise could. Building up a diverse workforce requires effort, intent, and paying attention to individuals in addition to statistics.
You’ve consistently been someone to advocate for DEI, both in the Kubernetes and engineering communities and in life, in general. We at Fairwinds are committed to these topics, too. Beyond basics, what next-level recommendations do you have for companies like us to continue to move DEI programs forward in a way that benefits both individuals and organizations?
I feel like I'm far from an expert in this area. Mostly, I try to listen to a diverse chorus of thoughtful voices. It's important to listen when people share their life and tech experiences generally, not only when they're explicitly talking about diversity. I've learned heaps from the writing, talks, and tweets of Stephen Augustus, Erica Joy Baker, Keirsten Brager, Liz Fong-Jones, Alice Goldfuss, Jam Leomi, Sage Sharp, and many others.
One thing I can confidently advise organizations is to support and embrace people from non-traditional backgrounds such as career-changers, bootcamp grads, and people with seemingly-irrelevant degrees. There are so many ways to solve any particular engineering problem, and people from non-traditional backgrounds can be more likely to offer unique answers that break down barriers. It's important to ensure your hiring practices don't accidentally push away these candidates, but it's equally important to actively support and sponsor them once you have them on your teams. Tech is strongly idiomatic, and great ideas can be passed over if they're not presented in the expected ways. Pay attention to this and make sure you're not missing out!
My confident advice to individuals is to read Kimberlé Crenshaw, the brilliant author who introduced the term "intersectionality". Her work is far more accessible than the phrase "critical theory" may lead you to believe. If you're building a more just world but not yet familiar with Crenshaw's writing, it'll change your life.
Fairwinds’ company values are respect, inclusion, compassion and kindness. We work every day to bring these values to life in our work and in our relationships with our colleagues, partners and customers. Do you have specific values that you try to bring to your work?
In my work, I try to always remember that tech is meaningless without people. In my life more generally, I strive to embody the principles of intersectional feminism, mutual aid, and leading by example. Attacking and defending computer systems is naturally joyful, and I'm amazingly lucky to be able to support myself that way. While I'm at it, I try to lift up others around me and become the role model I wished for myself.
People know you for asking the question, “I wonder what would happen if we…” Would you answer that question for us today??
To answer that question, first we'll have to ask it! Here's a couple examples:
"I wonder what would happen if we signed all our Kubernetes cluster CA certificates using our corporate internal root CA?" It may be fine, or it may be a disaster. If that sounds like your brand of fun, you'll really enjoy my KubeCon NA 2020 talkPKI the Wrong Way: Simple TLS Mistakes and Surprising Consequences.
"I wonder what would happen if we all treated each other with respect and helped each other develop our highest powers?" I'm not sure, but that sounds great and I'm doing my best to find out by trying!