Keeping your Kubernetes infrastructure up-to-date is critical for security, compliance, stability, and performance, and it usually gives your cluster feature enhancements as well. Forced upgrades can wreak havoc on your workloads and supporting infrastructure. Working with our team of site reliability engineers (SREs), we identified the eight most critical factors for a smooth Kubernetes upgrade. From deprecated APIs and breaking changes to add-on compatibility and beyond, this post provides a practical guide with key steps to ensure your upgrade is successful and minimizes disruption to your Kubernetes workloads.
1. Deprecated/Removed API Versions
Kubernetes is an API-driven system; for most upgrades, the API has deprecation and eventual removal of older versions. When an API version is deprecated, it remains functional for no more than nine months or three minor releases. This can cause significant impact to applications and the underlying infrastructure that depend on deprecated APIs.
Key Steps
- Audit Your Resources — regularly check your Kubernetes resources for deprecated API versions. Tools such as kubectl api-resources or kubectl api-versions will list available API versions and check the Kubernetes deprecation page for your specific version to identify which APIs are deprecated or removed, but automated tools may provide a more comprehensive analysis. For example, Fairwinds developed Pluto to find deprecated and removed APIs in Kubernetes infrastructure and to help you find the workloads that would be affected.
- Update Manifests and API endpoints — modify your Kubernetes manifests (YAML files) to use the new API versions and update any workloads that interact with the deprecated Kubernetes APIs.
- Key API Migrations — common migrations include:
- Ingress — From extensions/v1beta1 to networking.k8s.io/v1
- CronJob — From batch/v1beta1 to batch/v1
- CustomResourceDefinition — From apiextensions.k8s.io/v1beta1 to apiextensions.k8s.io/v1
- Update Third-Party Components — ensure all third-party tools and add-ons (more on this below) are compatible with the new API versions. Make sure that you check for updates to Helm charts, operators, CustomResourceDefinitions (CRDs), and your Kubernetes-related tools that you use to interact with the cluster.
- Gradual Migration — consider migrating resources in batches to minimize risk. Also consider using feature flags or canary deployments where possible.
- Validate and Monitor — after you complete your migration, monitor your cluster for any issues or anomalies. You can also use Kubernetes' deprecation insights and recommendations to identify any remaining deprecated API usage.
- Keep an Eye on Future Compatibility — stay informed about upcoming Kubernetes releases and their deprecation schedules. Implement a process for regular API version reviews and updates.
- Aim for No Rollbacks – once you’ve migrated your APIs, avoid rolling back to versions using deprecated APIs. Downgrades can be problematic and can lead to eroded trust for the orchestration system.
The exact migration process varies depending on your specific Kubernetes version and the APIs you're utilizing. Always refer to the official Kubernetes documentation and release notes for the most up-to-date information on API deprecations and migrations. The Kubernetes Deprecated API Migration Guide is a helpful resource for getting specific details on how to migrate your configurations accordingly.
2. Breaking Changes
Breaking changes occur when modifications to the Kubernetes API or behavior impacts existing functionality. These changes can lead to application failures if not addressed prior to upgrading.
Key Steps
- Review Release Notes — read the release notes (more on this below) for the target Kubernetes version and look for sections specifically mentioning breaking changes, deprecations, or “MAJOR” Semantic Versioning changes.
- API Changes — check for changes to API versions, as indicated above.
- Feature Gates — be aware of changes to feature gates, especially those moving to general availability (GA) or being removed, then adjust your configurations if you were relying on alpha or beta features.
- Behavioral Changes — look for modifications in how core components or resources behave, such as changes in scheduling behavior, networking, or storage.
- Default Values — check if any default values for configurations have changed and update your manifests if you were relying on previous defaults.
- Command-Line Tools — be aware that changes to kubectl or other official Kubernetes CLI tools can be required for the upgrade. Make sure you update your scripts or CI/CD pipelines tools, as they may be lacking compatibility with the new API endpoints.
- Add-ons and Extensions — verify compatibility of critical add-ons, such as CNI plugins, ingress controllers, or monitoring solutions. Plan for upgrades of these components (more on add-ons below).
- Custom Controllers and Operators — if you have custom controllers or operators, be sure they're compatible with the new version and test thoroughly, especially if they interact with changing APIs.
- Resource Requirements — check if there are changes to minimum resource requirements for nodes or control plane components and then plan for infrastructure upgrades if needed.
- Security Changes — be aware of any security-related changes, such as modifications to role-based access control or Pod Security Admission (or a third-party admission plugin) and adjust your security configurations accordingly.
- Workload Compatibility — test your applications in a staging environment with the new Kubernetes version and look for any unexpected behavior, especially in stateful applications.
- Upgrade Path — understand the supported upgrade paths; in some cases, you may need to upgrade through intermediate versions — and plan your upgrade strategy carefully, especially for multi-version jumps. See Semantic Versioning and release notes of all versions between your current version and the desired version.
3. Add-On Compatibility
Your Kubernetes infrastructure relies on various add-ons (such as monitoring tools, ingress controllers, and so on) that may also depend on specific API versions. Ensuring compatibility between these add-ons and the new Kubernetes version is vital for maintaining your system functionality, which is why Fairwinds created the open source tool GoNoGo. It checks K8s add-ons before an upgrade.
Key Steps
- Inventory Current Add-ons — create a list of all add-ons currently deployed in your cluster, including their versions. Note which ones are essential for your cluster's operation.
- Check Compatibility Documentation — review the documentation for each add-on to determine compatibility with the target Kubernetes version; you can often find compatibility matrices or version support information in the add-on's code repository, official website, or release notes.
- Version Compatibility Checks — use tools like the AWS command line interface (CLI) describe-addon-versions command to check compatibility for EKS add-ons. You’ll need to check the documentation for each add-on or use compatibility checking tools (if available).
- Plan for Updates — determine which add-ons need to be updated before or after the Kubernetes upgrade. Create an upgrade plan for each add-on that needs updating.
- Helm Chart Updates — for add-ons installed via Helm, check for updated chart versions that support the new Kubernetes version and upgrade these charts as part of your upgrade process.
4. Pod Disruption Budgets
Pod Disruption Budgets (PDBs) help manage voluntary disruptions in a cluster by specifying how many pods can be unavailable during maintenance activities or upgrades.
Key Steps
- Inventory Existing PDBs — make sure to understand their configurations and which applications they protect.
- Evaluate PDB Specifications — check minAvailable or maxUnavailable settings in PDBs to ensure they align with your upgrade strategy and consider using percentage-based values for flexibility during scaling. Modify PDBs if necessary to accommodate the upgrade process while maintaining application availability.
- Consider Application Requirements and Cluster Operations — ensure PDBs reflect the actual availability needs of your applications and align PDB settings with node drain operations during upgrades.
- Consider Workload Types — ensure PDBs are properly set for different workload types (Deployments, StatefulSets, and so on).
- Review Selector Labels — to make sure PDB selectors accurately match the pods they should protect.
- Maintenance scaling — assure your cluster’s cloud account has access to enough resources for the maintenance. Some cloud providers have resource quotas, which limit your account from accessing additional resources.
5. Versions of Configuration Tools
Configuration tools like Terraform or Helm must be compatible with the new Kubernetes version as well. Incompatible tools can lead to deployment failures or configuration issues.
Key Steps
- Inventory Current Tools — list all configuration management tools currently in use (such as Terraform, Helm, Ansible, Crossplane, and custom scripts) and note the current versions of these tools.
- Check Compatibility — review the documentation for each tool to ensure compatibility with the target Kubernetes version; look for version compatibility matrices or release notes that cover Kubernetes support.
- Plan for Tool Upgrades — identify which tools need to be updated before or after the Kubernetes upgrade, creating an upgrade plan for each tool that requires updating.
- Update Infrastructure as Code — review and update your IaC files (for example, Terraform configurations and Helm charts) to ensure they're compatible with both the new tool versions and the new Kubernetes version.
- Update CI/CD Pipelines — to use the updated tool versions, verifying that automated processes are compatible with the new Kubernetes version.
- Version Control — commit all changes to your version control system and tag your releases to mark pre- and post-upgrade states.
6. Review Kubernetes Readmes
Kubernetes provides extensive documentation and readmes that include important information about new features, deprecations, and best practices for each version upgrade.
Key Steps
- Reference Official Documentation — look at the official Kubernetes documentation for the version you're upgrading to and read the release notes and upgrade guides for your version jump.
- Review Version-Specific Changes — these are typically noted as "Notable Changes" or "Major Themes" sections in the release notes. Pay attention to deprecations, removals, and new features that might affect your cluster.
- Check Upgrade Prerequisites — this may include minimum versions of components or required configuration changes.
- Understand Upgrade Paths — verify the supported upgrade paths; Kubernetes typically only supports upgrades to the next minor version
("skipping MINOR versions when upgrading is unsupported").
- Review Component Versions — the compatibility of control plane components (including kube-apiserver, kube-controller-manager, and kube-scheduler). Make sure you understand the supported version skew between components (more on skew below).
- Check for Known Issues — review any known issues or caveats mentioned for the new version and look for workarounds or mitigation strategies if necessary.
- Prepare for Post-Upgrade Tasks — check whether there are any post-upgrade tasks mentioned, such as validating the cluster or updating manifests.
7. Validate Support Matrix
The support matrix outlines which products and services are compatible with specific versions of Kubernetes. Ensuring compatibility is important for maintaining your operational stability.
Key Steps
- Identify Current and Target Versions — determine your current Kubernetes version and the version you plan to upgrade to.
- Check Component Compatibility — review the compatibility of control plane components (that is, kube-apiserver, kube-controller-manager, and kube-scheduler) and make sure you understand the supported version skew between components.
- Verify Node Versions — make sure that your kubelet versions on worker nodes are compatible with the new control plane version; the kubelet version should not be newer than the API server version and can be up to two minor versions older.
- Review Add-on and Tool Compatibility — as outlined earlier in this post. Make updates as needed
- Consult Vendor-Specific Documentation — refer to the specific upgrade documentation and support matrices for managed Kubernetes services (AKS, EKS, and GKE).
- Review Upgrade Path and Check for Known Issues — make sure you know the supported upgrade paths for your specific Kubernetes distribution and look for known issues for the target version.
8. Validate Future Version Skew
Version skew refers to having different versions of control plane components (such as kube-apiserver) and worker nodes (kubelets). Managing this skew is essential for cluster stability.
Key Steps
- Understand the Version Skew Policy — Kubernetes has a specific supported version skew that defines the supported version differences between components. For example, in highly-availability (HA) clusters, the newest and oldest kube-apiserver instances must be within one minor version.
- Check Control Plane Components — make sure that all control plane components (kube-apiserver, kube-controller-manager, kube-scheduler) are compatible with the target version; plan for a gradual upgrade in HA clusters to maintain the supported version skew between kube-apiserver instances.
- Verify Node Versions — kubelet must not be newer than kube-apiserver and may be up to two minor versions older for Kubernetes versions before 1.28 (thereafter, kubelet may be three minor versions older than kube-apiserver).
- Consider kubectl Compatibility — kubectl is typically supported within one minor version (older or newer) of kube-apiserver.
- Plan Upgrade Order — make sure you understand the implications of version skew on the order in which components must be upgraded.
- Plan for Multiple Upgrades — if you’re upgrading across multiple versions, you may need to perform incremental upgrades to maintain compatibility.
Successful K8s Upgrades
Upgrading Kubernetes is not just a technical task; it requires you to plan carefully and consider multiple factors that can affect the performance and stability of your applications and services deployed on Kubernetes infrastructure. By focusing on these top eight aspects of upgrading Kubernetes, your organization can navigate upgrades more effectively and minimize potential disruptions to your K8s infrastructure.
There are a few things you need to do in every one of these areas:
- Stay informed about upcoming changes through official documentation and community channels.
- Test in a non-production environment to verify compatibility and functionality.
- Have a rollback plan in place in case of compatibility issues after the upgrade.
- Monitor your cluster and add-ons for any unexpected behavior or performance issues.
- Update your internal documentation and runbooks based on the changes you’re making.
- Identify any post-upgrade tasks mentioned in the documentation, such as updating manifests or validating the cluster.
- Plan to have a rule of thumb for what version of Kubernetes should be the desired version. For example, within two versions of the most recent version.
If you want to focus on your business differentiators in-house and leave the complexities of Kubernetes upgrades to an outside provider, consider Fairwinds Managed Kubernetes-as-a-Service.
