In the VMware 8 Tanzu environment, after upgrading to WCP 1.27, worker nodes and control plane nodes are created with the default operating system (Photon) even though Ubuntu was selected in the TKR selection.

This issue was observed only in existing Tanzu Kubernetes clusters. When the worker nodes are created with the wrong operating system, some system pods may remain in a pending state due to node selector labels not matching on the newly created worker nodes.

Workaround: Apply the node label with the Ubuntu version tag:

kubectl label node --overwrite run.tanzu.vmware.com/kubernetesDistributionVersion=v1.25.13---vmware.1-fips.1-tkg.1.ubuntu

To apply the label to all nodes:

kubectl get nodes | awk '{ print $1 }' | xargs -I {} kubectl label nodes {} run.tanzu.vmware.com/kubernetesDistributionVersion=v1.25.13---vmware.1-fips.1-tkg.1.ubuntu --overwrite

In vSphere version 8u3, Tanzu introduced a TKC annotation for the operating system:

run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu

We need to annotate the cluster with run.tanzu.vmware.com/resolve-os-image:

kubectl annotate tkc -n kubernetes-vns run.tanzu.vmware.com/resolve-os-image="os-name=ubuntu"