feat(registry): optionally push images to and mirror pulls through a local registry

[tdakkota]

Adds an opt-in local-registry path to the suite: E2E_REGISTRY=host:port makes it docker push the images it builds instead of kind loading them, and points every kind node's containerd at the same host as a pull-through mirror for docker.io, ghcr.io, quay.io and registry.k8s.io.

Default is unchanged. With E2E_REGISTRY unset, cfg.inRegistry leaves the image tags bare, build.BinaryImage takes the kind load branch exactly as before, configureRegistry() returns immediately, and nothing is written to /etc/containerd/certs.d. The only unconditional change is kind.yaml setting containerd's config_path to that directory — inert while the directory is empty. CI and anyone without a registry are unaffected.

Why this is worth having

On a machine that already runs a pull-through cache, a suite run stops hitting the internet for the kind node image, the dev etcd, and the operator's dependencies, and the built images move over the LAN rather than through kind load's tar-per-node side-load.

The hosts.toml is copied, not invented

It is modelled on the containerd config running on a real cluster — tdakkota/nixfra, modules/registry-mirrors.nix — because two of its details are load-bearing, non-obvious, and very easy to "simplify" back into a broken state. Both are asserted in internal/registry/registry_test.go so a future cleanup fails the build rather than the cluster.

1. docker.io's upstream must be https://registry-1.docker.io. https://docker.io is a website. containerd follows the fallback, gets HTML, and caches it in the content store as if it were a manifest. Every later pull of that image fails with unexpected media type text/html for sha256:..., and it stays broken across containerd restarts until the poisoned blob is evicted by hand:

   docker exec <node> ctr -n k8s.io content rm sha256:<digest>
   

2. override_path = true is required. The mirror is addressed as <registry>/v2/<domain>; without it containerd appends its own /v2/<repo> and requests .../v2/docker.io/v2/library/alpine/manifests/3.21, which 404s.

A third property is documented but not configurable: a pull-through mirror may re-digest what it caches (Zot serves an alpine index digest that differs from Docker Hub's), so pulls pinned to an upstream digest rather than a tag are inherently fragile through one.

Verification

Everything below ran against a Zot at 10.42.0.44:5000.

Full suite, role pools enabled, all four images pushed to and pulled from the registry:

Ran 11 of 11 Specs in 138.122 seconds
SUCCESS! -- 11 Passed | 0 Failed | 0 Pending | 0 Skipped

Kubelet confirms the pull path rather than a side-load:

Normal  Pulled  kubelet  Container image "10.42.0.44:5000/claude-e2e/oteldb:test-c986072a"
                         already present on machine and can be accessed by the pod

The exact hosts.toml this branch generates was then re-verified on a throwaway single-node cluster. Before writing any of it, the default path pulls fine; after writing it, a cold tag demonstrably routes through the mirror:

$ curl -s .../v2/docker.io/library/alpine/tags/list
{"tags":["3.19","3.20","3.21","3.22","3.23","latest"]}          <- before
$ docker exec <node> crictl pull docker.io/library/alpine:3.18
Image is up to date for sha256:802c91d5...
$ curl -s .../v2/docker.io/library/alpine/tags/list
{"tags":["3.18","3.19","3.20","3.21","3.22","3.23","latest"]}   <- after

quay.io/coreos/etcd:v3.5.17 (the dev etcd) and a registry-hosted claude-e2e/oteldb image both resolve through the same config.

Notes

• The registry is assumed to speak plain HTTP, so it must also be in the host docker daemon's insecure-registries — that is what the docker push and kind's own node-image pull go through. Documented in the README.
• manifests/cluster.yaml pins a bare spec.image; under a registry the suite patches the CR to the registry-qualified tag after apply, rather than forking the manifest.
• An explicitly host-qualified OTELDB_IMAGE=ghcr.io/... still wins over E2E_REGISTRY.
• Only the four registries an e2e run actually reaches are mirrored; nixfra's list is longer and is that infrastructure's concern.

🤖 Generated with Claude Code