feat: Phase 6 — push sources, upload path and Paper plugin

.superpowers/sdd/2026-08-09-phase-6-push/final-report.md

@@ -0,0 +1,117 @@
+# Phase 6 — Final report: closing the push path, tokens, and plugin/API drift
+
+## Status
+
+Done. All three loose ends (A, B, C) closed; spec brought up to date.
+
+## A — Push/upload wiring (`ingest`)
+
+`IngestConfig`/`IngestMain` now accept `push`/`upload` exactly like the pull sources:
+
+- `SUPPORTED_SOURCE_TYPES` gained `push`/`upload`; the stale "have no connector yet"
+  message was removed.
+- New shared env-var contract for both staged-source types (only one runs per job, so one
+  contract covers both): `APUS_SOURCE_STAGING_BUCKET` (required), `APUS_SOURCE_STAGING_ENDPOINT`,
+  `APUS_SOURCE_STAGING_PREFIX`, `APUS_SOURCE_STAGING_ACCESS_KEY`, `APUS_SOURCE_STAGING_SECRET_KEY`,
+  `APUS_SOURCE_STAGING_REGION` — documented in `ingest/README.md`.
+- `IngestMain.selectConnector` now returns `PushSourceConnector`/`UploadSourceConnector` for
+  those types.
+- New end-to-end test `PushIngestEndToEndTest` (`:ingest:integrationTest`, real MinIO via
+  Testcontainers): stages a zip in a staging prefix, runs `IngestMain.run` for both `push` and
+  `upload`, and asserts a valid bundle + manifest + region file land in the destination bucket.
+  This is the proof the push path now works end to end, not just that the connector classes work
+  in isolation.
+
+## B — Push-token generation (`operator`)
+
+Tokens are **tenant-scoped**, not per-`WorldSource` — the design spec already settled this
+("Service-Tokens sind mandantengebunden", §10.3), and the existing `FabricPushTokenRepository`
+already resolves a token to a *namespace*, not a source, which only makes sense under that
+reading.
+
+- New `PushTokenSecrets` (operator, package `tenant`) is the single canonical definition of the
+  Secret shape (label, data key, fixed name `apus-push-token`, `generate()` using `SecureRandom`
+  + URL-safe base64, 256 bits).
+- `TenantReconciler` creates this Secret once per tenant, alongside the namespace. Critically, it
+  is **never regenerated** on later reconciles (no `createOr(update)` here) — a fresh random
+  value on every resync would silently invalidate whatever `paper-worldpush` was already
+  configured with. Ownership is checked the same way every other tenant resource is (name+UID
+  labels), so a conflicting pre-existing Secret is refused rather than adopted.
+- `TenantStatus` gained `pushTokenSecret` (the Secret's fixed, non-secret *name* only). The token
+  value itself never appears in status, an event, or a log line.
+- `api`'s `FabricPushTokenRepository` now delegates its constants to `PushTokenSecrets` instead
+  of duplicating them (removes exactly the kind of drift risk this whole phase report is about).
+
+**RBAC, documented but not implemented as YAML** (no manifest/Helm/Kustomize infrastructure
+exists anywhere in this repo to hang it on): `FabricPushTokenRepository`'s Javadoc now spells out
+that its current `list()`-by-label lookup, unavoidably, needs `get`/`list` on **all** Secrets
+cluster-wide (Kubernetes RBAC cannot filter by label) — broader than ideal — and documents the
+concrete narrower alternative (enumerate tenants via the already-listable `Tenant` CR, then `get`
+the fixed-name Secret per namespace, letting RBAC restrict to `resourceNames: ["apus-push-token"]`
++ `get` only) as a deliberate follow-up, not implemented now to avoid an invasive rewrite of
+already-tested code under this task's scope. Flagged as a concern below and as open item 9 in the
+spec.
+
+## C — Plugin/API alignment (`paper-worldpush` ↔ `api`)
+
+Token transport was already consistent (path segment both sides; only `config.yml`'s comment
+wrongly said "bearer token" — fixed). The real break was the **request body**: the plugin sent
+`{tenant, worldName, fileCount, bytesUploaded}`, but `PushController`/`PushReportRequest` only
+ever deserializes `{sourceName, version}` — every real push report would have 400'd. Fixed:
+
+- New required `world-source-name` config key (`WorldPushConfig.sourceName()`) — the target
+  `push`-type `WorldSource`'s name, since a token alone is tenant-, not source-, scoped.
+- `PushCycleRunner` now generates a timestamp-style `version` per cycle (injectable `Clock` for
+  tests) and `PushSummary`/`HttpPushNotifier` send exactly `{"sourceName", "version"}` on the
+  wire.
+- New `HttpPushNotifierTest` (JDK `HttpServer` stub, matching this repo's established pattern)
+  locks in the correct path-segment token and JSON body shape.
+
+## Spec (`docs/superpowers/specs/2026-08-08-apus-design.md`)
+
+- New §0 "Stand der Umsetzung" at the top: all six phases built, sharding deliberately not built
+  (references §14 Phase 4), and the three open items (identity broker unselected, OIDC never
+  tested against a real broker, `paper-worldpush`'s save window untested).
+- §4 module table: Java 21 → 25 everywhere (root `build.gradle.kts` toolchain applies to every
+  subproject uniformly); `world-ingest`/`runner-image` → actual dir names `ingest`/`runner`;
+  added `hosting` (Dockerfile-only, no Gradle module) and corrected `api`/`ui` to their current
+  form; corrected `operator`'s stack (JOSDK + fabric8, no Micronaut).
+- §13.2: CRD-generation note marked done (describes the `crdgen` source set); test-coverage table
+  corrected per module, including the new push/upload/E2E tests and the two still-open gaps
+  (identity broker, Paper save window).
+- §15: items 1 (connector order) and 4 (CRD generation) marked resolved; item 2 (bucket
+  notifications) corrected — neither notifications nor polling was built, a direct completion
+  callback from the writer was, which is now documented; item 3 (identity broker) confirmed still
+  open; new items 8 (Paper save window untested) and 9 (push-token RBAC broader than ideal).
+
+## Verification
+
+- `./gradlew build -x :runner:test -x :operator:integrationTest -x :ingest:integrationTest -x :api:integrationTest` — green.
+- `:ingest:test` + `:ingest:integrationTest` — green, including the new `PushIngestEndToEndTest` (push and upload, parameterized).
+- `:operator:test` (incl. 5 new `TenantReconciler` push-token tests) + `:operator:integrationTest` — green.
+- `:api:test` + `:api:integrationTest` — green (existing `FabricPushTokenRepositoryTest`/`PushControllerTest` pass unchanged against the now-shared constants).
+- `:paper-worldpush:test` — green, including the new `HttpPushNotifierTest`.
+- `:runner:integrationTest` — **still red**, but not from this phase's work: `IngestRenderContractTest` was missing `APUS_BUNDLE_SOURCE_NAME` entirely (a required field since before phase 6; fixed as a drive-by) and, after that, fails a second, unrelated assertion — its hardcoded expected bucket-listing omits `level.dat`, which `BundleWriter` has included in every bundle for longer than this test's expectation has been stale. Pre-existing, unrelated to push/upload/tokens; left as a flagged concern rather than fixed under this task's scope.
+
+All started Testcontainers (MinIO, k3s) were torn down by the test framework itself; no
+containers were left running. No `isukuverlagcms-*` containers were touched.
+
+## Concerns
+
+- `runner:integrationTest`'s `IngestRenderContractTest` has a second, pre-existing failure
+  (stale expected bucket listing vs. `BundleWriter`'s actual `level.dat` inclusion) unrelated to
+  this phase — needs its own fix.
+- Push-token RBAC: the working implementation still needs cluster-wide Secret read for the api
+  ServiceAccount (see B above); the narrower `Tenant`-enumeration approach is documented but not
+  built.
+- No Kubernetes manifest/Helm/Kustomize directory exists anywhere in this repo — every RBAC
+  requirement found this phase (this one, and the pre-existing one `FabricPushTokenRepository`
+  already flagged) is documented in Javadoc only, with nothing to actually apply on a cluster.
+- The deeper shape mismatch between how `paper-worldpush` stages data (many individual raw region
+  files dropped incrementally under a prefix, no single "version" blob) and what
+  `AbstractStagedSourceConnector.fetch()` expects to read (one object at `prefix + version.id()`,
+  archive or raw) was not resolved — fixing the wire *request* makes the HTTP call succeed, but
+  the ingest job it triggers would still try to `getObject` a single key that was never written
+  this way. This is a real design gap between `paper-worldpush` and the `push` ingest connector,
+  bigger than the auth/wire-format alignment this task asked for; flagged for a dedicated design
+  pass rather than patched here.

.superpowers/sdd/2026-08-09-phase-6-push/task-2-report.md

@@ -0,0 +1,136 @@
+# Phase 6, Task 2 — Push/upload connectors and the upload+push API endpoints
+
+## Status
+
+Done. `./gradlew :api:test :ingest:test` passes (144 + 65 tests, 0 failures). All four
+Docker/MinIO-backed integration tests (excluded from the above, run via
+`./gradlew :api:integrationTest :ingest:integrationTest`) also pass, including the two that
+empirically probe the presigned-upload security properties this task cared about most.
+
+## What was built
+
+### 1. `PushSourceConnector` / `UploadSourceConnector` (`ingest/.../connector/`)
+
+Both extend a new `AbstractStagedSourceConnector`, which holds all the behaviour: `discover()`
+always returns an empty list (push semantics, per `WorldSourceConnector`'s own contract), and
+`fetch()` is functionally identical to `S3SourceConnector.fetch()` — get the object at
+`prefix + version.id()`, extract it if `Archives.isArchive` recognises the key's extension,
+otherwise copy it as a single raw file. Only `type()` differs between the two concrete classes
+(`"push"` / `"upload"`). Deliberately *not* refactored to share code with `S3SourceConnector`
+itself — that class already ships a passing test suite and touching it risked it for ~40 lines
+saved.
+
+Tests: `AbstractStagedSourceConnectorTest` (shared, real MinIO via Testcontainers, mirrors
+`S3SourceConnectorTest`'s own setup) is subclassed by `PushSourceConnectorTest` and
+`UploadSourceConnectorTest`, each proving `discover()` is always empty and `fetch()` correctly
+handles zip/tar.gz/raw staged objects. Excluded from `:ingest:test`, run via
+`:ingest:integrationTest` (Docker) — same convention as the existing `S3SourceConnectorTest`.
+
+**Not wired up**: `IngestConfig`/`IngestMain` (which select a connector by `APUS_SOURCE_TYPE`)
+still explicitly reject `"push"`/`"upload"` with *"The push sources ... have no connector yet"*.
+Those files live outside `ingest/.../connector/`, which the task brief named as the hard
+boundary — wiring them is left for whoever owns that file.
+
+### 2. `POST /api/uploads` + `POST /api/uploads/{uploadId}/complete` (`api/.../rest/upload/`)
+
+The design spec's §11.1 table lists only `POST /api/uploads`, with no completion endpoint
+documented anywhere. I added the `/complete` sub-resource anyway because without it the feature
+cannot do anything useful — S3 multipart uploads are not readable objects until
+`CompleteMultipartUpload` runs, and (see below) that call is deliberately *not* presigned, so
+something has to invoke it. This is the one place I went beyond the literal two-endpoint list;
+flagging it here rather than silently expanding scope.
+
+- `MultipartUploadService` does all the S3 work. `CreateMultipartUpload`, `ListParts`,
+  `CompleteMultipartUpload`, `AbortMultipartUpload` are all performed by the backend itself with
+  its own staging credentials — **never presigned**, even though `S3Presigner` can presign all
+  four. Only `UploadPart` is presigned and handed to the caller.
+- `stagingKey(prefix, namespace, sourceName, version, fileName)` is the single place an S3 key
+  is ever built, and it's a pure static function — unit-tested directly with adversarial inputs
+  (`version = "../../bluemap-globex/other-source"` etc.) proving the key can never leave
+  `<prefix>/<namespace>/<sourceName>/...`. `namespace` always comes from `TenantResolver` (JWT),
+  never the request body.
+- `StagingS3ClientFactory` provides the `S3Client`/`S3Presigner` beans against one shared,
+  platform-wide staging bucket (credentials via `@Value`, no hardcoded defaults, matching this
+  module's existing `LogSourceFactory` convention) — tenant isolation is entirely a matter of key
+  prefix, not separate buckets/credentials per tenant.
+
+### 3. `POST /api/push/{token}` (`api/.../rest/push/`)
+
+The one endpoint in the module that is **not** JWT-authenticated
+(`@Secured(SecurityRule.IS_ANONYMOUS)`, deliberate). Authentication is entirely
+`PushTokenRepository#resolveNamespace(token)`.
+
+- **Token storage**: neither `WorldSourceSpec` nor `TenantSpec` (both in `operator/`, out of
+  this task's scope) carry a token field, so `FabricPushTokenRepository` reads plain Kubernetes
+  `Secret`s instead — labelled `apus.onelitefeather.net/service-token: world-push`, living in the
+  tenant's own namespace, `data.token` holding the raw shared secret. This requires the API's
+  ServiceAccount to have cluster-wide `get`/`list` on Secrets carrying that label — an RBAC grant
+  outside this task's scope, documented in the class Javadoc as an exact contract for whoever
+  wires it up (a future operator reconciler, most likely).
+- **Constant-time, exhaustive comparison**: every candidate Secret is compared via
+  `MessageDigest.isEqual` (never `String.equals`/`Arrays.equals`, which short-circuit on the
+  first differing byte), and the loop never returns early on a match — scanning every candidate
+  every time, so neither a per-byte guess nor "how many secrets exist before this one" leaks
+  through timing.
+- Controller flow: resolve namespace from token first (before the body is even read) → validate
+  request → look up the named `WorldSource` **within that resolved namespace**, filtered to
+  `type == "push"` → for each of its configured worlds, create one `WorldIngest` (mirrors
+  `WorldSourceReconciler.triggerIngests`'s per-world loop for pull sources — same code path, per
+  design spec §6.4). Every failure before a valid, well-formed request is a uniform 404
+  (`NotFoundException`) — unknown token, valid token + unknown source, valid token + source of
+  the wrong type all look identical.
+
+## Which upload restrictions are actually enforced — the honest answer
+
+| Restriction | Status | How it was verified |
+|---|---|---|
+| **Confined to the caller's own tenant prefix** | **Enforced, structurally.** | `stagingKey` is a pure function of a server-derived namespace; unit-tested with adversarial input. S3 has no `..`-traversal semantics, so there is no string a caller can supply that escapes the prefix. |
+| **A presigned part URL can't be redirected to a different key** | **Enforced, confirmed against real MinIO.** | `MultipartUploadServiceIntegrationTest.aPresignedPartUrlCannotBeRedirectedToADifferentTenantsKey` swaps the tenant segment in a legitimate presigned URL and gets HTTP 403 from MinIO — SigV4 signs the exact key. |
+| **A part can't carry more bytes than it was sized for** | **Enforced, confirmed against real MinIO (2026-08-09).** | `Content-Length` is set on each presigned `UploadPartRequest`; AWS SDK v2 includes it among that URL's signed headers. Sending more bytes than declared gets HTTP 403 `SignatureDoesNotMatch` from MinIO before the extra bytes are accepted — I drove a real oversized `PUT` against a real MinIO instance rather than trusting SDK documentation (which does not state this explicitly). **Caveat**: verified against MinIO specifically, not independently re-verified against Ceph RGW (the actual production backend per design spec §9.1). Both implement SigV4 presigned-URL validation the same way, so I expect the same result, but that is an inference from one data point, not a second measurement. |
+| **Total upload size is capped** | **Enforced, but only at completion, and that's by design.** | Completion (`CompleteMultipartUpload`) is deliberately never presigned — the backend performs it itself, after summing every part's *real, S3-recorded* size via `ListParts` (never trusting anything the client claims) and comparing against `maxUploadBytes`. An oversized upload is aborted, never completed — confirmed by `completeUploadAbortsAndRejectsWhenTheActualUploadedTotalExceedsTheConfiguredMaximum`, which also confirms the upload is genuinely gone (`NoSuchUploadException` from `ListParts` afterward). Given the per-part `Content-Length` pinning above also holds, in practice a client cannot even get an oversized part accepted in the first place — but the `ListParts` check is what makes the limit a *guarantee* rather than a hope, independent of that per-part behavior. |
+| **Tenant's actual storage budget** | **Deliberately out of scope for this endpoint.** | Design spec §10.2 already establishes Ceph RGW's per-user quota as the real, application-independent backstop for `Tenant.spec.storage.quota`. `maxUploadBytes` here only bounds one absurd single upload, not the tenant's overall budget — that's Ceph's job regardless of anything this code does or gets wrong. |
+| **Short URL validity** | **Enforced by S3/MinIO itself.** | `X-Amz-Expires` in the presigned URL (default 900s, configurable), standard SigV4 behaviour — not specific to this implementation. |
+
+**Net assessment**: every restriction the task asked for turned out to be enforceable, and every
+one of the security-relevant ones was checked against a real S3-compatible backend rather than
+assumed from documentation — including the one I expected going in to be the weakest link
+(per-part size), which turned out to work. The one caveat worth carrying forward is Ceph RGW vs.
+MinIO for the `Content-Length`-pinning behaviour specifically.
+
+## Push token abuse cases tested
+
+`PushControllerTest` (in-memory fakes, no Docker) and `FabricPushTokenRepositoryTest` (real
+fabric8 mock Kubernetes API, `@EnableKubernetesMockClient`) together cover: unknown token, blank
+token, a valid token used to try to reach a source name that only exists in a *different*
+tenant's namespace, a valid token whose resolved source is not of type `push`, a source with no
+configured worlds, missing request fields, and — the core property — that a token valid for one
+namespace never creates a `WorldIngest` in another. `UploadControllerTest` covers the equivalent
+set for the JWT-authenticated `/api/uploads` path (viewer role rejected, foreign-tenant source
+name not found, wrong-type source not found, missing fields), plus a wiring proof that a
+request passing every controller-level check really does reach `MultipartUploadService`.
+
+## Concerns / follow-ups for whoever picks this up next
+
+- **RBAC for `FabricPushTokenRepository`**: the API's ServiceAccount needs cluster-wide
+  `get`/`list` on Secrets labelled `apus.onelitefeather.net/service-token`. Not part of this
+  task's `ingest/`+`api/` scope; needs a ClusterRole/ClusterRoleBinding somewhere in the
+  deployment manifests.
+- **Nothing creates the push-token Secret yet.** A platform-admin/tenant-owner (or, eventually,
+  an operator reconciler) needs to actually create `Secret`s matching the documented shape — see
+  `FabricPushTokenRepository`'s Javadoc for the exact contract.
+- **`IngestConfig`/`IngestMain` still reject `push`/`upload`.** The connectors exist and are
+  tested but aren't reachable from a real ingest job until that file (outside this task's scope)
+  is updated.
+- I could not find a documented completion endpoint for `upload` in the design spec at all — see
+  "went beyond the literal two-endpoint list" above. Worth a deliberate design decision rather
+  than inheriting mine by default.
+
+## File-restriction compliance
+
+Kept to `ingest/src/.../connector/`, `api/src/...`, and their tests, with one deliberate
+exception: `settings.gradle.kts` (added the AWS SDK version catalog entry — already used
+project-wide) and `api/build.gradle.kts`/`ingest/build.gradle.kts` (added the AWS SDK/S3-presigner
+and Testcontainers-MinIO dependencies, and the `*IntegrationTest`/`*ConnectorTest` exclude/include
+lines for the new Docker-backed tests). None of these are reachable without touching a build file
+outside the strict directory list; all three are shared, module-level config, not
+`paper-worldpush/`, and I did not touch anything under `operator/` or `paper-worldpush/`.