From 51ce6edbc08ecc1278192f474b34206ef2a807b3 Mon Sep 17 00:00:00 2001 From: Jason Morrow Date: Thu, 27 Aug 2026 11:56:10 -0400 Subject: [PATCH] Add aws-iam-authenticator so the runner can drive a remote cluster The runner already resolves its client from a kubeconfig when one is present, so targeting another cluster was always possible in principle. The image just had no way to authenticate to EKS: it is alpine plus the Go binary, with no credential helper. Adds aws-iam-authenticator, pinned by version and per-arch SHA-256. Chosen over the AWS CLI, which has no musl build, and over the AWS SDK in Go, which would put AWS specifics into a binary that otherwise has none. This way EKS support is a property of the image and the runner keeps speaking plain kubeconfig. Motivation is the agent sandbox platform: the control plane and the workloads it schedules are different trust levels, and separate clusters is a stronger boundary than separate namespaces. It is also a prerequisite for dispatching across cells, where a controller confined to its own cluster cannot address more than one. Verified: both arches build, the binary reports 0.7.20 and exposes `token -i`, and a tampered checksum fails the build rather than silently installing something else. docs/cross-cluster.md covers the kubeconfig shape, the two IAM pieces, and the gotcha that an unmapped identity returns 401 rather than 403. --- Dockerfile | 23 +++++++++++ docs/cross-cluster.md | 93 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 docs/cross-cluster.md diff --git a/Dockerfile b/Dockerfile index 9c47a57..f1877ae 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,9 +1,32 @@ FROM golang:alpine@sha256:3ad57304ad93bbec8548a0437ad9e06a455660655d9af011d58b993f6f615648 AS build RUN apk --no-cache add ca-certificates +# aws-iam-authenticator lets a kubeconfig's exec credential plugin mint EKS +# tokens, which is what allows the runner to drive a cluster it is not running +# in. See docs/cross-cluster.md. +# +# A standalone static binary rather than the AWS CLI, which has no musl build, +# and rather than the AWS SDK in Go, which would make a deliberately +# cloud-agnostic binary AWS-aware. This way EKS support is a property of the +# image and the runner keeps speaking plain kubeconfig. +FROM alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b AS authenticator +ARG TARGETARCH +ARG AWS_IAM_AUTHENTICATOR_VERSION=0.7.20 +RUN apk --no-cache add curl && \ + case "${TARGETARCH}" in \ + amd64) sha=a60921d38d2b51bc998f37eb1eba81c92aed21d9315a35a01b283ac398c43f87 ;; \ + arm64) sha=8afd45da2ba288ee515e31215ab5a7247d92449c179281e0610522226c5e21a5 ;; \ + *) echo "unsupported TARGETARCH=${TARGETARCH}" >&2; exit 1 ;; \ + esac && \ + curl -fsSL -o /aws-iam-authenticator \ + "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${AWS_IAM_AUTHENTICATOR_VERSION}/aws-iam-authenticator_${AWS_IAM_AUTHENTICATOR_VERSION}_linux_${TARGETARCH}" && \ + echo "${sha} /aws-iam-authenticator" | sha256sum -c - && \ + chmod +x /aws-iam-authenticator + FROM alpine:latest@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b ARG TARGETPLATFORM # copy the ca-certificate.crt from the build stage COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ +COPY --from=authenticator /aws-iam-authenticator /usr/local/bin/aws-iam-authenticator COPY ${TARGETPLATFORM}/opslevel-runner /opslevel-runner ENTRYPOINT ["/opslevel-runner"] diff --git a/docs/cross-cluster.md b/docs/cross-cluster.md new file mode 100644 index 0000000..13a4bc8 --- /dev/null +++ b/docs/cross-cluster.md @@ -0,0 +1,93 @@ +# Driving a cluster the runner does not run in + +The runner resolves its Kubernetes client with +`clientcmd.NewNonInteractiveDeferredLoadingClientConfig` — a kubeconfig if +`$KUBECONFIG` or `~/.kube/config` is present, otherwise in-cluster config. So +targeting a remote cluster has always been possible in principle; the image just +had no way to authenticate to EKS. + +`aws-iam-authenticator` is now in the image for exactly that. The runner itself +stays cloud-agnostic — it speaks kubeconfig, and EKS support is a property of +the image rather than of the binary. That was the reason for a standalone +authenticator over either the AWS CLI (no musl build) or the AWS SDK in Go +(which would put AWS specifics in a tool that otherwise has none). + +## Why you would want this + +The control plane and the workloads it schedules are different trust levels. The +runner holds an OpsLevel API token and has pod-create rights; job pods run +tenant-supplied work. Putting them in separate clusters is a stronger boundary +than separate namespaces, and it is what the agent sandbox platform's +architecture assumes — the controller schedules work but never executes tenant +content. + +It is also a prerequisite for dispatching across cells later: a controller that +can only create pods in its own cluster cannot address more than one. + +## Configuring it + +Mount a kubeconfig and point `KUBECONFIG` at it. **The kubeconfig holds no +credentials**, so a ConfigMap is fine — SOPS is not needed: + +```yaml +apiVersion: v1 +kind: Config +current-context: target +clusters: + - name: target + cluster: + server: https://XXXXXXXX.gr7.us-east-1.eks.amazonaws.com + certificate-authority-data: +contexts: + - name: target + context: { cluster: target, user: target } +users: + - name: target + user: + exec: + apiVersion: client.authentication.k8s.io/v1beta1 + command: aws-iam-authenticator + args: ["token", "-i", ""] +``` + +AWS credentials come from the pod's environment. On EKS that means Pod Identity +or IRSA on the ServiceAccount the runner runs as, in the cluster it runs *in*. +`aws-iam-authenticator` needs no IAM permissions of its own — `token` presigns +an STS `GetCallerIdentity` request locally and makes no AWS API call. It only +needs an identity to sign as. + +## The two IAM pieces + +**In the cluster the runner runs in:** a Pod Identity association binding the +runner's ServiceAccount to an IAM role. The role needs no policies attached. + +**In the cluster the runner targets:** an EKS access entry for that role. + +Scope the access entry to a namespace, not the cluster. An entry with +`kubernetes_groups` bound to a Role in the target namespace is enough for the +runner's work — pods, pods/exec, pods/log, configmaps — and is a much better +fit than `AmazonEKSClusterAdminPolicy`. Least privilege matters more when the +credential lives in a different cluster than the one it opens. + +## Gotchas + +**An unmapped identity gets 401, not 403.** With +`authentication_mode = "API"`, a principal with no access entry fails +authentication rather than authorization, so it reads as a credentials problem +and sends you looking at the wrong layer. + +**Verify Pod Identity reaches the authenticator.** Pod Identity supplies +credentials through `AWS_CONTAINER_CREDENTIALS_FULL_URI` and +`AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`. The authenticator's SDK supports that, +but it is worth confirming in your deployment rather than assuming — the failure +is an opaque token error at startup. + +**`AWS_REGION` must be set** if the target cluster is not in the runner's +default region. + +## Updating the authenticator + +Pinned by version and SHA-256 in the `Dockerfile`, with per-architecture +checksums from the release's `authenticator__checksums.txt`. Both must +be updated together; a mismatch fails the build rather than silently installing +something unexpected.