Kubernetes Version Upgrade AKS Setup

Instructions

In order to conduct upgrades of AKS clusters, we will require an additional credential for the use of Azure APIs. This is different from the credentials for Cloudnatix installation or VM-monitoring.

Create a custom role

First, create a custom role.

SUBSCRIPTION_ID=<your subscription ID>
ROLE_NAME=<name for the custom role. e.g. "AKS upgrader">

az role definition create --role-definition "{
  \"Name\": \"${ROLE_NAME}\",
  \"Description\": \"Perform upgrades of AKS clusters\",
  \"Actions\": [
    \"Microsoft.ContainerService/managedClusters/read\",
    \"Microsoft.ContainerService/managedClusters/write\",
    \"Microsoft.ContainerService/managedClusters/agentPools/read\",
    \"Microsoft.ContainerService/managedClusters/agentPools/write\"
  ],
  \"AssignableScopes\": [\"/subscriptions/${SUBSCRIPTION_ID}\"]
}"

Set up a new managed identity

Create a new managed identity

Next, create a new managed identity. It needs to reside within a resource group; any resource group should be fine. You could create your own or reuse an existing one.

RESOURCE_GROUP=<identity resource group>
LOCATION=<location of identity, e.g. "eastus">
IDENTITY_NAME=<name, e.g. "aks-upgrade-controller">

az identity create --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --location $LOCATION --subscription $SUBSCRIPTION_ID

Assign roles to the identity

Then set up the role assignment to this identity. The role name should be the one you created above. This assigns the created role for the access of API changes, and “Log Analytics Contributor” to keep track of the update. Since AKS upgrade is a long-running process, this access is required to wait for the process to be completed.

NOTE: Sometimes there may be propagation delay after creating the role above that causes the assignment here to fail. If the command says that the role does not exist, wait a few minutes and try again.

IDENTITY_CLIENT_ID=$(az identity show -g ${RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)

az role assignment create --role ${ROLE_NAME} --assignee $IDENTITY_CLIENT_ID --scope "/subscriptions/${SUBSCRIPTION_ID}"
az role assignment create --role "Log Analytics Contributor" --assignee $IDENTITY_CLIENT_ID --scope "/subscriptions/${SUBSCRIPTION_ID}"

If you want to limit the API access to a certain list of resource groups, here would be the flow. Keep in mind that “Log Analytics Contributor” still needs to be in the scope of the entire subscription.

IDENTITY_CLIENT_ID=$(az identity show -g ${RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)

for CLUSTER_RESOURCE_GROUP in rg1 rg2 rg3...; do
   az role assignment create --role ${ROLE_NAME} --assignee $IDENTITY_CLIENT_ID --scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${CLUSTER_RESOURCE_GROUP}"
done
az role assignment create --role "Log Analytics Contributor" --assignee $IDENTITY_CLIENT_ID --scope "/subscriptions/${SUBSCRIPTION_ID}"

Allow access to the identity

Finally, allow CloudNatix to call those Azure APIs with this identity. The easiest way to do this is to set up the workload identity federation.

FEDERATION_NAME=<name of the federated credential, e.g. "cloudnatix-aks-upgrader">
az identity federated-credential create --name $FEDERATION_NAME --identity-name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --issuer 'https://global-controller-kops-irsa.s3.us-west-2.amazonaws.com/us-west-2.prod' --subject 'system:serviceaccount:k8s-upgrade-controller:k8s-upgrade-controller-processor' --audiences amazonaws.com

After that, using the client ID of the identity, the subscription ID, and the tenant ID run the following command to create the credential in CloudNatix.

cnatix k8s upgrade credentials create --csp azure --subscriptionid <Subscription ID> --tenantid <Tenant ID> --clientid <Client ID>

Now your clusters should be ready to upgrade. There can be a replication delay in creating the federated credential done above that will cause an error in the “Cluster Credential integration” check performed by CloudNatix. If this happens, wait a few minutes and try again.