Kubernetes Version Upgrade Assistant

Overview

CloudNatix offers extensive CLI tooling for upgrading Kubernetes versions in CSP managed environments such as EKS and AKS. To use this functionality using the cnatix CLI tool call:

cnatix k8s upgrade

This will give you an overview of all of the commands. They can be roughly categorized into:

Setup
Readiness
Automated Upgrade Deployment
Post Upgrade

Setup and Permissions

To utilize the CloudNatix upgrade tool, there are a few steps to be taken depending on your setup.

For EKS resources, please use the Kubernetes Version Upgrade EKS Setup page. For AKS resources, please use the Kubernetes Version Upgrade AKS Setup page.

Notification Webhooks (optional)

If you want CloudNatix to notify you on the status of your upgrades you can use webhooks to receive updates on Slack and Teams. To do this, you can obtain your webhook URL and run the following command:

# Slack
cnatix k8s upgrade notification-configs create --slack-web-hook-url <webhook url>

# Teams
cnatix k8s upgrade notification-configs create --teams-web-hook-url <webhook url>

For Slack you can also use the app-token, channel-id, and token with their corresponding arguments (--slack-<...>).

Readiness

CloudNatix can help you identify potential issues you may face when trying to upgrade your clusters. The two main commands for this are

cnatix k8s upgrade supported-versions
cnatix k8s upgrade readiness

supported-versions will help you determine what K8s versions your CSP supports via the --csp flag (along with --region for azure). You can also use this for a given cluster using the --cluster flag.

cnatix k8s upgrade readiness will perform many checks on your clusters to determine if they are ready to be upgraded. These checks include PDB configuration checks, Deprecated-API checks, Cluster state checks, Add-on compatibility checks, and K8s version specific checks that are updated for each new release.

Some common usages of the readiness command are

# check the readiness of a cluster to a specific version
cnatix k8s upgrade readiness --cluster <Cluster> --version <K8s version>

# check the readiness of a list of clusters to a specific version
cnatix k8s upgrade readiness --cluster <Cluster 1>,...,<Cluster n> --version <K8s version>

Automatic Upgrade Deployment

After ensuring that your cluster is ready to be upgraded to a given K8s version using the readiness command, you can also use the CloudNatix upgrade tool to take action and upgrade your cluster. CloudNatix will create and perform an action plan to upgrade your cluster to the target K8s version while respecting PDBs, and automatically handling node drains. You can also run the command in --dry-run mode to view the upgrade path.

Some example usage:

# Start an upgrade
cnatix k8s upgrade start --cluster <Cluster> --version <K8s version> 

# Start a dry-run that ignores the cnatix readiness check
cnatix k8s upgrade start --cluster <Cluster> --version <K8s version> --dry-run --force

# Start an upgrade that ignores the cnatix readiness check and will fail if a pod becomes unready after a node group upgrade
cnatix k8s upgrade start --cluster <Cluster> --version <K8s version> --force --fail-on-unready-pod

Working with an upgrade in-flight

If you want to perform operations on an in-flight upgrade, CloudNatix enables you to perform operations such as:

# Cancel
cnatix k8s upgrade cancel --cluster <Cluster>

# Pause
cnatix k8s upgrade pause --cluster <Cluster>

# Resume
cnatix k8s upgrade resume --clsuter <Cluster>

# Retry a failed upgrade
cnatix k8s upgrade retry --cluster <Cluster>

# Check the status of an in-flight upgrade
cnatix k8s upgrade status --cluster <Cluster> --verbose

Upgrade scheduling

If you want to schedule an upgrade to start hands free at a later time, you can do this throught the schedule command:

# Schedule an upgrade for some time in the future (contains the same options as start)
cnatix k8s upgrade schedule --cluster <Cluster> --start-time <YYYY-MM-DD hh:mm:ss> --deadline <YYYY-MM-DD hh:mm:ss>

This command acts as a delayed cnatix k8s upgrade start command, allowing you to schedule upgrades for low traffic times without the need for active triggering.

Post Upgrade checks

Once an upgrade has been complete, a series of post-flight checks are run and included in the status of the upgrade. However you can run these checks manually using the following command.

cnatix k8s upgrade postflight --cluster <Cluster>

In addition to a single post-flight check, you can get a summary of all upgrades run recently using the command

cnatix k8s upgrade summary

The summary command will give information about the number of upgrades that have succeeded and failed, as well as categorizing clusters into common reasons for upgrade failure so you can easily see where issues may exist in your clusters.

Summary

CloudNatix provides a wide variety of tools to assist and automate the process of upgrade your Kubernetes clusters. By using the Readiness command on clusters that you are targeting to upgrade, you can easily find what configurations need to change before upgrading instead of manually searching for deprecated APIs for each version upgrade.

The tooling also vastly decreases both the time it takes to upgrade clusters as well as the chance for human mistakes by automating the node drain and creation. By easing the entire process, CloudNatix helps organizations stay on top of security and performance upgrades that come with new K8s version multiple times per year.