As we outlined in our first post in this series, there are some things that you should simply never, ever do in Kubernetes. Corey Quinn, Chief Cloud Economist at The Duckbill Group, had a lively conversation with Kendall Miller, President of Fairwinds, and Stevie Caldwell, Senior Site Reliability Engineer at Fairwinds discussing a few things that development and operations teams should never ever do in Kubernetes if they want to get the most out of the leading container orchestrator. If you’re looking to maximize Kubernetes efficiency, here’s what to look out for.
(Remember, this is never should you ever, so the headlines might seem a bit odd, a bit obvious, or even surprising!)
This K8s efficiency mistake and the next one are really closely related, and what it really comes down to when putting workloads in “Texas-size houses” is blast radius — that is, how much of what you’re running will be impacted if (or when) something goes wrong. If you have a few big workloads running and one of them goes down, a lot of things can be affected. So while you can put your workloads in huge “houses,” you also need to have the workload be aware of these things. In other words, know what will be impacted if your workload goes down. And don’t forget to consider what’s required to get everything back up and running again if everything does go down.
One of the things that people struggle with most when they're deploying workloads on Kubernetes is understanding what their applications need. Making the decisions about what kind of nodes or instance types are best for what they're running is a challenge, and so is knowing what kind of resource request to make and what resources to allocate to their applications. A lot of people start out allocating way too much, and then you have more nodes or larger nodes in your cluster than you need. Kubernetes schedules your workload based on what you request, so if you request 200 Millicore and you only use eight, nothing else will get scheduled that would conflict with that and you’ll have far more headroom than you need for your applications. In this case you might be able to run your applications effectively, but you’ll be losing out on the efficiency of your Kubernetes deployment.
Often we suggest that starting small is a good development practice as you're learning and getting started with Kubernetes. At the beginning, it makes sense to start out by building things in constrained environments, where you can always scale up if you need to. The important thing to remember is that this is a crutch that you should only need as you get started; you’ll eventually run out of room to grow on that plan.
Keep in mind, though, that there's a difference between starting out small and taking something that clearly needs significantly more computing power and sticking it in the smallest possible space. If you do that, it's always going to be running at 100% CPU and it's still not going to be working. While it’s true that there's no one size that fits everything you’re going to do, you’re not going to save money or frustration by putting your workloads somewhere that they can’t work effectively or efficiently.
Don't use houses that are too big or you'll be sacrificing efficiency (and $$) and don't use houses that are too small or you could over-utilize a node, causing a cascade of failures due to resource starvation.
There are a lot of people using cloud and using cloud native technologies, like Kubernetes, that make things like scaling easy. And yet, they’re not auto scaling. That's one of the biggest promises of the cloud. Don't move to Kubernetes and turn off auto scaling. It's really good at autoscaling, and something you should take advantage of as part of your use of Kubernetes.
People tend to be very interested in implementing auto scaling to scale up, which is important because it keeps your application online when there’s a surge in resource needs. Unfortunately, people always forget to figure out how to scale down sensibly, because the failure mode there is just spending a bit of extra money. Over the last year, though, many environments saw user traffic fall off a cliff while the infrastructure cost remained a straight line. Why? Because they never actually tested scaling down and so it didn’t happen. Don’t ignore autoscaling, but also take the time to make sure your environment doesn't explode if half of its nodes disappear. Autoscaling can certainly improve your efficiency, so take the time to implement and test it, and never, ever ignore it.
Watch the entirely entertaining webinar on demand to learn what else you should never, ever do in Kubernetes.
