Upgrade to Kubernetes 1.28

[dduportal]

Per https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar, Microsoft stops supporting Kubernetes 1.27 in July 2024.

As such, we must upgrade our Kubernetes clusters to 1.28.

Last issue for 1.27 was: #3948. Since last issue, we got rid of AWS and DigitalOcean clusters: we only have 4 Azure AKS clusters now.

---

Task list:

• Upgrade kubectl within packer-images All-in-One
  • First upgrade the updatecli manifest (chore(updatecli) track 1.28.x line of kubectl versions packer-images#1239)
  • Then apply the updatecli PR that is produced by this change: (Bump kubectl version to 1.28.11 packer-images#1240)
  • Trigger a release of the packer-images (https://github.com/jenkins-infra/packer-images/releases/tag/1.73.1)
  • Then apply the Kubernetes-Management PR that will use this new version -(Bump agent templates for infra.ci.jenkins.io (packer-image 1.73.1) kubernetes-management#5331)
  • Also apply the puppet (jenkins-infra/jenkins-infra) PR with the new version - (Bump Packer Agent Templates (All-In-One) Version to 1.73.1 jenkins-infra#3485)
    --
• Check AKS changelog for 1.28 - https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2023-11-05
• Upgrade AKS cluster infracijio-agents-1 (details)
• Upgrade AKS cluster cijenkinsio-agents-1 (details)
• Upgrade AKS cluster privatek8s (details)
• Upgrade AKS cluster publick8s (details)
• Add Logo ;)
• Open issue for next Kubernetes version

[smerle33]

infracijio-agents-1

• Announcement: no announcement needed as the cluster is not used yet
• Disable kubernetes management for infracijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - feat(infracijioagents1): removing temporarily from management to upgrade to kubernetes 1.28 kubernetes-management#5337
• Terraform PR to upgrade cluster to 1.28 but NO merge - feat(infracijenkinsio_agents_1): upgrade to kubernetes 1.28 azure#738
  • checks that all checks are ok
  • Merge the PR
• Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - Revert "feat(infracijioagents1): removing temporarily from management to upgrade to kubernetes 1.28" kubernetes-management#5338
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1247/

🎉

[smerle33]

cijioagents1

We plan to handle this upgrade Tuesday 25 of june morning Paris time

• Announcements:
  • status.jenkins.io - open issue for kub 1.28 upgrade for cijioagent1 status#508 - @dduportal and @smerle33
  • developer mailing list + jenkins-infra mailing - @dduportal and @smerle33
  • IRC #jenkins-infra - @dduportal and @smerle33
  • Add a banner on ci.jenkins.io - @dduportal and @smerle33
• Disable kubernetes management for cijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - feat(cijioagents1): removing temporarily from management to upgrade to Kubernetes 1.28 kubernetes-management#5348 - @jayfranco999
• Disable the 2 clouds defined in ci.jenkins.io on puppet (jenkins-infra/jenkins-infra) for the operation ci.jenkins.io-agents-1-bom and ci.jenkins.io-agents-1 - feat(cijio_agents_1): Disabling kubernetes for cluster cijio-agents1 jenkins-infra#3495 - @jayfranco999
• Terraform PR to upgrade cluster to 1.28.9 but NO merge - feat(cijio_agents_1): upgrade to kubernetes 1.28 azure#750 - @jayfranco999
  • checks that all checks are ok
  • Merge the PR
• Re-Enable the 2 clouds puppet defined - Revert "feat(cijio_agents_1): Disabling kubernetes for cluster cijio-agents1" jenkins-infra#3496 - @jayfranco999
• Try the connexion to the Clouds for this cluster within the ci.jenkins.io UI
• Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - - @jayfranco999 and @smerle33
• Check terraform job and eventually trigger it for success with no changes - - @jayfranco999 and @smerle33
• Close Annoucements - @jayfranco999 and @smerle33 and @dduportal
  • status.jenkins.io - close cijioagents1 closing upgrade kub 1.28 status#509
  • developer mailing list + jenkins-infra mailing - @dduportal and @smerle33
  • IRC #jenkins-infra - @dduportal and @smerle33
  • Remove the banner on ci.jenkins.io - @dduportal and @smerle33

---

POSTMORTEM

we need to ensure that puppet catch up the change to disable and to re-enable the cloud cluster. The disable is double handled by the banner in ci.jenkins.io but still ...

[smerle33]

AZURE

Update: AKS Upgrade plan privatek8s

privatek8s

• Announcement: date/time UTC
  • Open datetime announcement on status.jenkins.io - open issue for kub 1.28 upgrade for privatek8s status#512
  • Jenkins-infra mailing lists - https://groups.google.com/g/jenkins-infra/c/0s44tR8EnXQ
  • IRC #jenkins-infra
• Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - feat(privatek8s): Removing temporarily from management to upgrade to kub 1.28 kubernetes-management#5353
• Terraform PR to upgrade cluster to 1.28 - feat(privatek8s): upgrade to kubernetes 1.28 azure#756
• Merge Terraform PR and let the upgrade happen
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1270/
• Enable kubernetes management for privatek8s cluster and check it runs successfully - Revert "feat(privatek8s): Removing temporarily from management to upgrade to kub 1.28" kubernetes-management#5362
• Check for services healthiness (pagerduty, datadog, manual control, logging in on infra.ci.jenkins.io/release.ci.jenkins.io with the Infra Agents Health job https://release.ci.jenkins.io/job/Infra%20Agents%20Health/)
• Close announcements

[smerle33]

Update on publick8s Azure AKS upgrade: plan Thursday 4 March at 07h00 UTC

• Changelog for Kube 1.28 to be checked with a focus on kubernetes/kubernetes@master/CHANGELOG/CHANGELOG-1.28.md#urgent-upgrade-notes
• Changelog focus on Azure : https://github.com/Azure/AKS/blob/master/CHANGELOG.md?rgh-link-date=2024-06-18T11%3A54%3A32Z#release-2023-11-05
• Announcement: Thursday 4 March at 07h00 UTC
  • status.jenkins.io - Open issue for publick8 azure cluster upgrade to kub-1.28 status#520 - @jayfranco999
  • developer mailing list (Authentication outage due to LDAP restart) + jenkins-infra mailing - https://groups.google.com/g/jenkinsci-dev/c/8ia38jYOWbI - @dduportal
  • #jenkins-infra Matrix IM channel - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$adCzjRUx0gTGRByCrzpoCdHEuywDSbHvUsAxwPjWxfc?via=g4v.dev&via=gitter.im&via=matrix.org - @smerle33
• Disable kubernetes-management for publick8s cluster - feat(publick8s): Removing temporarily from management to upgrade to kub 1.28 kubernetes-management#5385 - @jayfranco999
• Bump Kube version to 1.28 for publick8s - feat(publick8s): upgrade to kubernetes 1.28 azure#772 - @jayfranco999
• Terraform jenkins-infra/azure project has a successful build without changes on the main branch after the upgrades - - @jayfranco999 @dduportal @smerle33
• Check the LDAP restart successfully once upgrade is performed. If the pod restarted and is running, try the service by logging-in on ci.jenkins.io and repo.jenkins-ci.org. @jayfranco999 @dduportal @smerle33
• Check the Artifact Caching Proxy ("ACP") service in azure/publick8s is up and running use this link https://repo.azure.jenkins.io/public/org/jenkins-ci/plugins/plugin/4.61/ @jayfranco999 @dduportal @smerle33
• Enable kubernetes-management back for publick8s - Revert "feat(publick8s): Removing temporarily from management to upgrade to kub 1.28" kubernetes-management#5387 - @jayfranco999 and check there are no warning in the helmfile log output (NOT the checks!) related to Kubernetes resources
• Check the pagerduty alerts are all gone
• Close announcements
  • status.jenkins.io - closed issue for publick8s cluster as it is upgraded to kub 1.28 status#521 - @jayfranco999
  • developer mailing list (Authentication outage due to LDAP restart) + jenkins-infra mailing - - @jayfranco999
  • #jenkins-infra Matrix IM channel - @smerle33

[jayfranco999]