[Release-1.32] - Fallback to default registry endpoint is broken when using "*" wildcard mirror in registries.yaml with containerd 2.0

[brandond]

Backport fix for Fallback to default registry endpoint is broken when using "*" wildcard mirror in registries.yaml with containerd 2.0

• Fallback to default registry endpoint is broken when using "*" wildcard mirror in registries.yaml with containerd 2.0 #11857

[mdrahman-suse]

Validation on RC v1.32.3-rc2+k3s1

• registries.yaml

$ cat /etc/rancher/k3s/registries.yaml
---
mirrors:
  "*":
    endpoint:
      - "example.com"

Testing

• Copy registries.yaml
• Install k3s
• Ensure cluster comes up

Replication

$ k3s -v
k3s version v1.32.2+k3s1 (381620ef)
go version go1.23.6

• Pods in ContainerCreating state

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   coredns-ff8999cc5-wj622                   0/1     ContainerCreating   0          2m50s
kube-system   helm-install-traefik-crd-v7l2n            0/1     ContainerCreating   0          2m51s
kube-system   helm-install-traefik-f6wpt                0/1     ContainerCreating   0          2m51s
kube-system   local-path-provisioner-774c6665dc-82w6w   0/1     ContainerCreating   0          2m50s
kube-system   metrics-server-6f4c6675d5-xjn2c           0/1     ContainerCreating   0          2m50s

• Error related to image pull in logs

Mar 19 23:41:48 k3s[8436]: E0319 23:41:48.349230    8436 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"metrics-server-6f4c6675d5-xjn2c_kube-system(11aad0fd-7caf-4147-8e3d-fc99b3ba5957)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"metrics-server-6f4c6675d5-xjn2c_kube-system(11aad0fd-7caf-4147-8e3d-fc99b3ba5957)\\\": rpc error: code = NotFound desc = failed to start sandbox \\\"dbaa9de5d5b291010ce4d186f65971b1502cb88a891498385b1850cc48faab76\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": docker.io/rancher/mirrored-pause:3.6: not found\"" pod="kube-system/metrics-server-6f4c6675d5-xjn2c" podUID="11aad0fd-7caf-4147-8e3d-fc99b3ba5957"

Validation

$ k3s -v
k3s version v1.32.3-rc2+k3s1 (f5921ea6)
go version go1.23.6

• Pods in Running/Completed state

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-ff8999cc5-dz4hp                   1/1     Running     0          69s
kube-system   helm-install-traefik-crd-gkg6w            0/1     Completed   0          70s
kube-system   helm-install-traefik-gtx75                0/1     Completed   2          70s
kube-system   local-path-provisioner-774c6665dc-xnx2z   1/1     Running     0          69s
kube-system   metrics-server-6f4c6675d5-lwz5q           1/1     Running     0          69s
kube-system   svclb-traefik-b2010dd3-5qvl5              2/2     Running     0          21s
kube-system   traefik-67bfb46dcb-k258p                  1/1     Running     0          21s

• No error related to image pull in logs