upgrade-kind-images Issue

[SgtCoDFish]

The kind image for k8s 1.32.0 has an incorrect SHASUM for arm64 currently.

This was found in trust-manager, which is using this PR's kind images: cert-manager/trust-manager#509

Specifically, this file.

$ make test-smoke
[info]: downloaded /Users/ashley.davis/workspace/cert-manager-trust-manager/_bin/downloaded/tools/[email protected]_darwin_arm64
❌  cluster trust-manager not found. Starting ...
diff <(echo "sha256:4487ecad11a7bba166ab8781c5f5e914f33fb13215c6cacf8a9ce93e33c56923  -" | cut -d: -f2) <(/Users/ashley.davis/workspace/cert-manager-trust-manager/_bin/tools/crane manifest --platform=linux/arm64 docker.io/kindest/node:v1.32.0 | sha256sum)
1c1
< 4487ecad11a7bba166ab8781c5f5e914f33fb13215c6cacf8a9ce93e33c56923  -
---
> eff24f9d99bc56271a456484d87cd6e6fc0beec7d4418958d589804703c00588  -
make: *** [_bin/downloaded/containers/arm64/docker.io/kindest/node+v1.32.0@sha256+4487ecad11a7bba166ab8781c5f5e914f33fb13215c6cacf8a9ce93e33c56923.tar] Error 1

The main bug is that running make upgrade-kind-images on the latest commit (8efc8e4) of the makefile-modules repo does not fix this issue - it produces no changes.

[maelvls]

looks like make upgrade-kind-images works as expected: it finds the sha256 from the GitHub Release's markdown contents:

curl -fsSL "https://api.github.com/repos/kubernetes-sigs/kind/releases/tags/v0.25.0" | jq '
  [ .body  | capture("- v?1\\.(?<minor>[0-9]+)(.(?<patch>[0-9]+))?: `kindest/node:v(?<version>[^@]+)@sha256:(?<sha256>[^`]+)`\r"; "g") ]
  | group_by(.minor) | map(max_by(.patch))
  | sort_by(.minor)'

returns

[
  # ...
  {
    "minor": "32",
    "patch": "0",
    "version": "1.32.0",
    "sha256": "2458b423d635d7b01637cac2d6de7e1c1dca1148a2ba2e90975e214ca849e7cb"
  }
]

I would let the Kind maintainers know that the list of checksums listed in https://github.com/kubernetes-sigs/kind/releases/tag/v0.25.0 needs updating

Alternatively, if we see this problem again and again, we could fetch the checksums ourselves instead of relying on the GitHub release's markdown contents? Maybe something like:

curl -fsSL "https://api.github.com/repos/kubernetes-sigs/kind/releases/tags/v0.25.0" | jq '
  [ .body  | capture("- v?1\\.(?<minor>[0-9]+)(.(?<patch>[0-9]+))?: `kindest/node:v(?<version>[^@]+)@.*`\r"; "g") ]
  | group_by(.minor) | map(max_by(.patch))
  | sort_by(.minor) | .[]' --compact-output \
  | while read -r line; do
    version=$(jq <<<"$line" -r '.version')
    sha256=$(crane manifest docker.io/kindest/node:v$version | sha256sum \
      | tee /dev/stderr \
      | awk '{print $1}')
    jq <<<"$line" ".sha256 = \"$sha256\""
  done

[SgtCoDFish]

Thanks for looking into it! I guess we're kinda in trouble here then. This has propagated to our projects and broken CI already. We probably need to manually override in makefile-modules here

Alternatively, if we see this problem again and again, we could fetch the checksums ourselves instead of relying on the GitHub release's markdown contents?

We can't do this unfortunately - Kind documents that the only supported way to get the checksums is from the release notes.

[maelvls]

ah, yes:

NOTE: You must use the @sha256 digest to guarantee an image built for this release, until such a time as we switch to a different tagging scheme. Even then we will highly encourage digest pinning for security and reproducibility reasons.

makes sense, let's revert 3976d24 ASAP manually override to avoid further downstream complications

[maelvls]

After manually overriding this, we could have a fix in modules/kind/kind-image-preload.mk:
since the GitHub release notes are the source of truth, we probably should most probably fetch the arch-specific manifests using the SHA-256 checksum rather than just the tag name...

makefile-modules/modules/kind/kind-image-preload.mk

Lines 35 to 46 in 8efc8e4

	# Download the images as tarballs. We must use the tag because the digest
	# will change after we docker import the image. The tag is the only way to
	# reference the image after it has been imported. Before downloading the
	# image, we check that the provided digest matches the digest of the image
	# that we are about to pull.
	$(images_tars): $(images_tar_dir)/%.tar: | $(NEEDS_CRANE)
	@$(eval image=$(subst +,:,$*))
	@$(eval image_without_digest=$(shell cut -d@ -f1 <<<"$(image)"))
	@$(eval digest=$(subst $(image_without_digest)@,,$(image)))
	@mkdir -p $(dir $@)
	diff <(echo "$(digest) -" | cut -d: -f2) <($(CRANE) manifest --platform=linux/$(HOST_ARCH) $(image_without_digest) | sha256sum)
	$(CRANE) pull $(image_without_digest) $@ --platform=linux/$(HOST_ARCH)

Before downloading the image, we check that the provided digest matches the digest of the image that we are about to pull.

Why are we doing this? Especially since Kind regularly overrides its tags