Skip to content

Add aws-iam-authenticator so the runner can drive a remote cluster - #323

Open
jasonopslevel wants to merge 1 commit into
mainfrom
add-eks-credential-helper
Open

Add aws-iam-authenticator so the runner can drive a remote cluster#323
jasonopslevel wants to merge 1 commit into
mainfrom
add-eks-credential-helper

Conversation

@jasonopslevel

Copy link
Copy Markdown
Contributor

Why

The runner already resolves its client from a kubeconfig when $KUBECONFIG is
set (clientcmd.NewNonInteractiveDeferredLoadingClientConfig), falling back to
in-cluster config otherwise. So driving a cluster it doesn't run in was always
possible in principle — the image just had no way to authenticate to EKS. It's
alpine plus the Go binary, with no credential helper, so a kubeconfig exec
plugin has nothing to exec.

Motivation is the agent sandbox platform. The control plane and the workloads it
schedules are different trust levels: the runner holds an OpsLevel API token and
pod-create rights, job pods run tenant-supplied work. Separate clusters is a
stronger boundary than separate namespaces. It's also a prerequisite for
dispatching across cells later — a controller confined to its own cluster can't
address more than one.

What

Adds aws-iam-authenticator 0.7.20, pinned by version and per-architecture
SHA-256 from the release's published checksums.

Chosen over the alternatives deliberately:

  • AWS CLI v2 has no musl build, so it isn't a one-line addition to an alpine
    image.
  • AWS SDK in Go would mint tokens natively, but go.mod currently has no
    AWS dependency at all. That would make a deliberately cloud-agnostic binary
    AWS-aware.

This way EKS support is a property of the image, and the runner keeps speaking
plain kubeconfig — the same image works against any cluster whose kubeconfig you
supply.

Verified

  • Both amd64 and arm64 authenticator stages build
  • Binary reports {"Version":"0.7.20"} and exposes token -i
  • A tampered checksum fails the build rather than silently installing
    something else — checked by mutating the SHA and confirming
    sha256sum: WARNING: 1 of 1 computed checksums did NOT match

Notes for reviewers

The kubeconfig carries no credentials — a ConfigMap is fine, no secret needed.
AWS credentials come from Pod Identity or IRSA on the ServiceAccount in the
cluster the runner runs in. aws-iam-authenticator token presigns an STS
request locally and makes no AWS API call, so the role needs no policies.

docs/cross-cluster.md covers the kubeconfig shape, the two IAM pieces, and the
gotcha that an unmapped identity gets 401, not 403 under
authentication_mode = "API" — which reads as a credentials problem and sends
you looking at the wrong layer.

The runner already resolves its client from a kubeconfig when one is
present, so targeting another cluster was always possible in principle.
The image just had no way to authenticate to EKS: it is alpine plus the
Go binary, with no credential helper.

Adds aws-iam-authenticator, pinned by version and per-arch SHA-256.
Chosen over the AWS CLI, which has no musl build, and over the AWS SDK in
Go, which would put AWS specifics into a binary that otherwise has none.
This way EKS support is a property of the image and the runner keeps
speaking plain kubeconfig.

Motivation is the agent sandbox platform: the control plane and the
workloads it schedules are different trust levels, and separate clusters
is a stronger boundary than separate namespaces. It is also a
prerequisite for dispatching across cells, where a controller confined to
its own cluster cannot address more than one.

Verified: both arches build, the binary reports 0.7.20 and exposes
`token -i`, and a tampered checksum fails the build rather than silently
installing something else.

docs/cross-cluster.md covers the kubeconfig shape, the two IAM pieces,
and the gotcha that an unmapped identity returns 401 rather than 403.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant