diff --git a/.superpowers/sdd/2026-08-09-phase-6-push/final-report.md b/.superpowers/sdd/2026-08-09-phase-6-push/final-report.md new file mode 100644 index 0000000..856c4eb --- /dev/null +++ b/.superpowers/sdd/2026-08-09-phase-6-push/final-report.md @@ -0,0 +1,117 @@ +# Phase 6 — Final report: closing the push path, tokens, and plugin/API drift + +## Status + +Done. All three loose ends (A, B, C) closed; spec brought up to date. + +## A — Push/upload wiring (`ingest`) + +`IngestConfig`/`IngestMain` now accept `push`/`upload` exactly like the pull sources: + +- `SUPPORTED_SOURCE_TYPES` gained `push`/`upload`; the stale "have no connector yet" + message was removed. +- New shared env-var contract for both staged-source types (only one runs per job, so one + contract covers both): `APUS_SOURCE_STAGING_BUCKET` (required), `APUS_SOURCE_STAGING_ENDPOINT`, + `APUS_SOURCE_STAGING_PREFIX`, `APUS_SOURCE_STAGING_ACCESS_KEY`, `APUS_SOURCE_STAGING_SECRET_KEY`, + `APUS_SOURCE_STAGING_REGION` — documented in `ingest/README.md`. +- `IngestMain.selectConnector` now returns `PushSourceConnector`/`UploadSourceConnector` for + those types. +- New end-to-end test `PushIngestEndToEndTest` (`:ingest:integrationTest`, real MinIO via + Testcontainers): stages a zip in a staging prefix, runs `IngestMain.run` for both `push` and + `upload`, and asserts a valid bundle + manifest + region file land in the destination bucket. + This is the proof the push path now works end to end, not just that the connector classes work + in isolation. + +## B — Push-token generation (`operator`) + +Tokens are **tenant-scoped**, not per-`WorldSource` — the design spec already settled this +("Service-Tokens sind mandantengebunden", §10.3), and the existing `FabricPushTokenRepository` +already resolves a token to a *namespace*, not a source, which only makes sense under that +reading. + +- New `PushTokenSecrets` (operator, package `tenant`) is the single canonical definition of the + Secret shape (label, data key, fixed name `apus-push-token`, `generate()` using `SecureRandom` + + URL-safe base64, 256 bits). +- `TenantReconciler` creates this Secret once per tenant, alongside the namespace. Critically, it + is **never regenerated** on later reconciles (no `createOr(update)` here) — a fresh random + value on every resync would silently invalidate whatever `paper-worldpush` was already + configured with. Ownership is checked the same way every other tenant resource is (name+UID + labels), so a conflicting pre-existing Secret is refused rather than adopted. +- `TenantStatus` gained `pushTokenSecret` (the Secret's fixed, non-secret *name* only). The token + value itself never appears in status, an event, or a log line. +- `api`'s `FabricPushTokenRepository` now delegates its constants to `PushTokenSecrets` instead + of duplicating them (removes exactly the kind of drift risk this whole phase report is about). + +**RBAC, documented but not implemented as YAML** (no manifest/Helm/Kustomize infrastructure +exists anywhere in this repo to hang it on): `FabricPushTokenRepository`'s Javadoc now spells out +that its current `list()`-by-label lookup, unavoidably, needs `get`/`list` on **all** Secrets +cluster-wide (Kubernetes RBAC cannot filter by label) — broader than ideal — and documents the +concrete narrower alternative (enumerate tenants via the already-listable `Tenant` CR, then `get` +the fixed-name Secret per namespace, letting RBAC restrict to `resourceNames: ["apus-push-token"]` ++ `get` only) as a deliberate follow-up, not implemented now to avoid an invasive rewrite of +already-tested code under this task's scope. Flagged as a concern below and as open item 9 in the +spec. + +## C — Plugin/API alignment (`paper-worldpush` ↔ `api`) + +Token transport was already consistent (path segment both sides; only `config.yml`'s comment +wrongly said "bearer token" — fixed). The real break was the **request body**: the plugin sent +`{tenant, worldName, fileCount, bytesUploaded}`, but `PushController`/`PushReportRequest` only +ever deserializes `{sourceName, version}` — every real push report would have 400'd. Fixed: + +- New required `world-source-name` config key (`WorldPushConfig.sourceName()`) — the target + `push`-type `WorldSource`'s name, since a token alone is tenant-, not source-, scoped. +- `PushCycleRunner` now generates a timestamp-style `version` per cycle (injectable `Clock` for + tests) and `PushSummary`/`HttpPushNotifier` send exactly `{"sourceName", "version"}` on the + wire. +- New `HttpPushNotifierTest` (JDK `HttpServer` stub, matching this repo's established pattern) + locks in the correct path-segment token and JSON body shape. + +## Spec (`docs/superpowers/specs/2026-08-08-apus-design.md`) + +- New §0 "Stand der Umsetzung" at the top: all six phases built, sharding deliberately not built + (references §14 Phase 4), and the three open items (identity broker unselected, OIDC never + tested against a real broker, `paper-worldpush`'s save window untested). +- §4 module table: Java 21 → 25 everywhere (root `build.gradle.kts` toolchain applies to every + subproject uniformly); `world-ingest`/`runner-image` → actual dir names `ingest`/`runner`; + added `hosting` (Dockerfile-only, no Gradle module) and corrected `api`/`ui` to their current + form; corrected `operator`'s stack (JOSDK + fabric8, no Micronaut). +- §13.2: CRD-generation note marked done (describes the `crdgen` source set); test-coverage table + corrected per module, including the new push/upload/E2E tests and the two still-open gaps + (identity broker, Paper save window). +- §15: items 1 (connector order) and 4 (CRD generation) marked resolved; item 2 (bucket + notifications) corrected — neither notifications nor polling was built, a direct completion + callback from the writer was, which is now documented; item 3 (identity broker) confirmed still + open; new items 8 (Paper save window untested) and 9 (push-token RBAC broader than ideal). + +## Verification + +- `./gradlew build -x :runner:test -x :operator:integrationTest -x :ingest:integrationTest -x :api:integrationTest` — green. +- `:ingest:test` + `:ingest:integrationTest` — green, including the new `PushIngestEndToEndTest` (push and upload, parameterized). +- `:operator:test` (incl. 5 new `TenantReconciler` push-token tests) + `:operator:integrationTest` — green. +- `:api:test` + `:api:integrationTest` — green (existing `FabricPushTokenRepositoryTest`/`PushControllerTest` pass unchanged against the now-shared constants). +- `:paper-worldpush:test` — green, including the new `HttpPushNotifierTest`. +- `:runner:integrationTest` — **still red**, but not from this phase's work: `IngestRenderContractTest` was missing `APUS_BUNDLE_SOURCE_NAME` entirely (a required field since before phase 6; fixed as a drive-by) and, after that, fails a second, unrelated assertion — its hardcoded expected bucket-listing omits `level.dat`, which `BundleWriter` has included in every bundle for longer than this test's expectation has been stale. Pre-existing, unrelated to push/upload/tokens; left as a flagged concern rather than fixed under this task's scope. + +All started Testcontainers (MinIO, k3s) were torn down by the test framework itself; no +containers were left running. No `isukuverlagcms-*` containers were touched. + +## Concerns + +- `runner:integrationTest`'s `IngestRenderContractTest` has a second, pre-existing failure + (stale expected bucket listing vs. `BundleWriter`'s actual `level.dat` inclusion) unrelated to + this phase — needs its own fix. +- Push-token RBAC: the working implementation still needs cluster-wide Secret read for the api + ServiceAccount (see B above); the narrower `Tenant`-enumeration approach is documented but not + built. +- No Kubernetes manifest/Helm/Kustomize directory exists anywhere in this repo — every RBAC + requirement found this phase (this one, and the pre-existing one `FabricPushTokenRepository` + already flagged) is documented in Javadoc only, with nothing to actually apply on a cluster. +- The deeper shape mismatch between how `paper-worldpush` stages data (many individual raw region + files dropped incrementally under a prefix, no single "version" blob) and what + `AbstractStagedSourceConnector.fetch()` expects to read (one object at `prefix + version.id()`, + archive or raw) was not resolved — fixing the wire *request* makes the HTTP call succeed, but + the ingest job it triggers would still try to `getObject` a single key that was never written + this way. This is a real design gap between `paper-worldpush` and the `push` ingest connector, + bigger than the auth/wire-format alignment this task asked for; flagged for a dedicated design + pass rather than patched here. diff --git a/.superpowers/sdd/2026-08-09-phase-6-push/task-2-report.md b/.superpowers/sdd/2026-08-09-phase-6-push/task-2-report.md new file mode 100644 index 0000000..98c4b3f --- /dev/null +++ b/.superpowers/sdd/2026-08-09-phase-6-push/task-2-report.md @@ -0,0 +1,136 @@ +# Phase 6, Task 2 — Push/upload connectors and the upload+push API endpoints + +## Status + +Done. `./gradlew :api:test :ingest:test` passes (144 + 65 tests, 0 failures). All four +Docker/MinIO-backed integration tests (excluded from the above, run via +`./gradlew :api:integrationTest :ingest:integrationTest`) also pass, including the two that +empirically probe the presigned-upload security properties this task cared about most. + +## What was built + +### 1. `PushSourceConnector` / `UploadSourceConnector` (`ingest/.../connector/`) + +Both extend a new `AbstractStagedSourceConnector`, which holds all the behaviour: `discover()` +always returns an empty list (push semantics, per `WorldSourceConnector`'s own contract), and +`fetch()` is functionally identical to `S3SourceConnector.fetch()` — get the object at +`prefix + version.id()`, extract it if `Archives.isArchive` recognises the key's extension, +otherwise copy it as a single raw file. Only `type()` differs between the two concrete classes +(`"push"` / `"upload"`). Deliberately *not* refactored to share code with `S3SourceConnector` +itself — that class already ships a passing test suite and touching it risked it for ~40 lines +saved. + +Tests: `AbstractStagedSourceConnectorTest` (shared, real MinIO via Testcontainers, mirrors +`S3SourceConnectorTest`'s own setup) is subclassed by `PushSourceConnectorTest` and +`UploadSourceConnectorTest`, each proving `discover()` is always empty and `fetch()` correctly +handles zip/tar.gz/raw staged objects. Excluded from `:ingest:test`, run via +`:ingest:integrationTest` (Docker) — same convention as the existing `S3SourceConnectorTest`. + +**Not wired up**: `IngestConfig`/`IngestMain` (which select a connector by `APUS_SOURCE_TYPE`) +still explicitly reject `"push"`/`"upload"` with *"The push sources ... have no connector yet"*. +Those files live outside `ingest/.../connector/`, which the task brief named as the hard +boundary — wiring them is left for whoever owns that file. + +### 2. `POST /api/uploads` + `POST /api/uploads/{uploadId}/complete` (`api/.../rest/upload/`) + +The design spec's §11.1 table lists only `POST /api/uploads`, with no completion endpoint +documented anywhere. I added the `/complete` sub-resource anyway because without it the feature +cannot do anything useful — S3 multipart uploads are not readable objects until +`CompleteMultipartUpload` runs, and (see below) that call is deliberately *not* presigned, so +something has to invoke it. This is the one place I went beyond the literal two-endpoint list; +flagging it here rather than silently expanding scope. + +- `MultipartUploadService` does all the S3 work. `CreateMultipartUpload`, `ListParts`, + `CompleteMultipartUpload`, `AbortMultipartUpload` are all performed by the backend itself with + its own staging credentials — **never presigned**, even though `S3Presigner` can presign all + four. Only `UploadPart` is presigned and handed to the caller. +- `stagingKey(prefix, namespace, sourceName, version, fileName)` is the single place an S3 key + is ever built, and it's a pure static function — unit-tested directly with adversarial inputs + (`version = "../../bluemap-globex/other-source"` etc.) proving the key can never leave + `///...`. `namespace` always comes from `TenantResolver` (JWT), + never the request body. +- `StagingS3ClientFactory` provides the `S3Client`/`S3Presigner` beans against one shared, + platform-wide staging bucket (credentials via `@Value`, no hardcoded defaults, matching this + module's existing `LogSourceFactory` convention) — tenant isolation is entirely a matter of key + prefix, not separate buckets/credentials per tenant. + +### 3. `POST /api/push/{token}` (`api/.../rest/push/`) + +The one endpoint in the module that is **not** JWT-authenticated +(`@Secured(SecurityRule.IS_ANONYMOUS)`, deliberate). Authentication is entirely +`PushTokenRepository#resolveNamespace(token)`. + +- **Token storage**: neither `WorldSourceSpec` nor `TenantSpec` (both in `operator/`, out of + this task's scope) carry a token field, so `FabricPushTokenRepository` reads plain Kubernetes + `Secret`s instead — labelled `apus.onelitefeather.net/service-token: world-push`, living in the + tenant's own namespace, `data.token` holding the raw shared secret. This requires the API's + ServiceAccount to have cluster-wide `get`/`list` on Secrets carrying that label — an RBAC grant + outside this task's scope, documented in the class Javadoc as an exact contract for whoever + wires it up (a future operator reconciler, most likely). +- **Constant-time, exhaustive comparison**: every candidate Secret is compared via + `MessageDigest.isEqual` (never `String.equals`/`Arrays.equals`, which short-circuit on the + first differing byte), and the loop never returns early on a match — scanning every candidate + every time, so neither a per-byte guess nor "how many secrets exist before this one" leaks + through timing. +- Controller flow: resolve namespace from token first (before the body is even read) → validate + request → look up the named `WorldSource` **within that resolved namespace**, filtered to + `type == "push"` → for each of its configured worlds, create one `WorldIngest` (mirrors + `WorldSourceReconciler.triggerIngests`'s per-world loop for pull sources — same code path, per + design spec §6.4). Every failure before a valid, well-formed request is a uniform 404 + (`NotFoundException`) — unknown token, valid token + unknown source, valid token + source of + the wrong type all look identical. + +## Which upload restrictions are actually enforced — the honest answer + +| Restriction | Status | How it was verified | +|---|---|---| +| **Confined to the caller's own tenant prefix** | **Enforced, structurally.** | `stagingKey` is a pure function of a server-derived namespace; unit-tested with adversarial input. S3 has no `..`-traversal semantics, so there is no string a caller can supply that escapes the prefix. | +| **A presigned part URL can't be redirected to a different key** | **Enforced, confirmed against real MinIO.** | `MultipartUploadServiceIntegrationTest.aPresignedPartUrlCannotBeRedirectedToADifferentTenantsKey` swaps the tenant segment in a legitimate presigned URL and gets HTTP 403 from MinIO — SigV4 signs the exact key. | +| **A part can't carry more bytes than it was sized for** | **Enforced, confirmed against real MinIO (2026-08-09).** | `Content-Length` is set on each presigned `UploadPartRequest`; AWS SDK v2 includes it among that URL's signed headers. Sending more bytes than declared gets HTTP 403 `SignatureDoesNotMatch` from MinIO before the extra bytes are accepted — I drove a real oversized `PUT` against a real MinIO instance rather than trusting SDK documentation (which does not state this explicitly). **Caveat**: verified against MinIO specifically, not independently re-verified against Ceph RGW (the actual production backend per design spec §9.1). Both implement SigV4 presigned-URL validation the same way, so I expect the same result, but that is an inference from one data point, not a second measurement. | +| **Total upload size is capped** | **Enforced, but only at completion, and that's by design.** | Completion (`CompleteMultipartUpload`) is deliberately never presigned — the backend performs it itself, after summing every part's *real, S3-recorded* size via `ListParts` (never trusting anything the client claims) and comparing against `maxUploadBytes`. An oversized upload is aborted, never completed — confirmed by `completeUploadAbortsAndRejectsWhenTheActualUploadedTotalExceedsTheConfiguredMaximum`, which also confirms the upload is genuinely gone (`NoSuchUploadException` from `ListParts` afterward). Given the per-part `Content-Length` pinning above also holds, in practice a client cannot even get an oversized part accepted in the first place — but the `ListParts` check is what makes the limit a *guarantee* rather than a hope, independent of that per-part behavior. | +| **Tenant's actual storage budget** | **Deliberately out of scope for this endpoint.** | Design spec §10.2 already establishes Ceph RGW's per-user quota as the real, application-independent backstop for `Tenant.spec.storage.quota`. `maxUploadBytes` here only bounds one absurd single upload, not the tenant's overall budget — that's Ceph's job regardless of anything this code does or gets wrong. | +| **Short URL validity** | **Enforced by S3/MinIO itself.** | `X-Amz-Expires` in the presigned URL (default 900s, configurable), standard SigV4 behaviour — not specific to this implementation. | + +**Net assessment**: every restriction the task asked for turned out to be enforceable, and every +one of the security-relevant ones was checked against a real S3-compatible backend rather than +assumed from documentation — including the one I expected going in to be the weakest link +(per-part size), which turned out to work. The one caveat worth carrying forward is Ceph RGW vs. +MinIO for the `Content-Length`-pinning behaviour specifically. + +## Push token abuse cases tested + +`PushControllerTest` (in-memory fakes, no Docker) and `FabricPushTokenRepositoryTest` (real +fabric8 mock Kubernetes API, `@EnableKubernetesMockClient`) together cover: unknown token, blank +token, a valid token used to try to reach a source name that only exists in a *different* +tenant's namespace, a valid token whose resolved source is not of type `push`, a source with no +configured worlds, missing request fields, and — the core property — that a token valid for one +namespace never creates a `WorldIngest` in another. `UploadControllerTest` covers the equivalent +set for the JWT-authenticated `/api/uploads` path (viewer role rejected, foreign-tenant source +name not found, wrong-type source not found, missing fields), plus a wiring proof that a +request passing every controller-level check really does reach `MultipartUploadService`. + +## Concerns / follow-ups for whoever picks this up next + +- **RBAC for `FabricPushTokenRepository`**: the API's ServiceAccount needs cluster-wide + `get`/`list` on Secrets labelled `apus.onelitefeather.net/service-token`. Not part of this + task's `ingest/`+`api/` scope; needs a ClusterRole/ClusterRoleBinding somewhere in the + deployment manifests. +- **Nothing creates the push-token Secret yet.** A platform-admin/tenant-owner (or, eventually, + an operator reconciler) needs to actually create `Secret`s matching the documented shape — see + `FabricPushTokenRepository`'s Javadoc for the exact contract. +- **`IngestConfig`/`IngestMain` still reject `push`/`upload`.** The connectors exist and are + tested but aren't reachable from a real ingest job until that file (outside this task's scope) + is updated. +- I could not find a documented completion endpoint for `upload` in the design spec at all — see + "went beyond the literal two-endpoint list" above. Worth a deliberate design decision rather + than inheriting mine by default. + +## File-restriction compliance + +Kept to `ingest/src/.../connector/`, `api/src/...`, and their tests, with one deliberate +exception: `settings.gradle.kts` (added the AWS SDK version catalog entry — already used +project-wide) and `api/build.gradle.kts`/`ingest/build.gradle.kts` (added the AWS SDK/S3-presigner +and Testcontainers-MinIO dependencies, and the `*IntegrationTest`/`*ConnectorTest` exclude/include +lines for the new Docker-backed tests). None of these are reachable without touching a build file +outside the strict directory list; all three are shared, module-level config, not +`paper-worldpush/`, and I did not touch anything under `operator/` or `paper-worldpush/`. diff --git a/api/build.gradle.kts b/api/build.gradle.kts index 683e121..9739e26 100644 --- a/api/build.gradle.kts +++ b/api/build.gradle.kts @@ -40,6 +40,15 @@ dependencies { annotationProcessor(platform(libs.micronaut.serde.bom)) annotationProcessor(libs.micronaut.serde.processor) + // AWS SDK v2 -- see settings.gradle.kts for why this SDK family. Backs both this module's own + // authenticated staging-bucket calls (CreateMultipartUpload, ListParts, + // CompleteMultipartUpload/AbortMultipartUpload -- see MultipartUploadService's Javadoc for why + // those specifically are never presigned) and the presigned UploadPart URLs POST /api/uploads + // hands back to the caller (design spec §11.1) via S3Presigner, which ships inside this same + // `s3` artifact (see settings.gradle.kts). + implementation(platform(libs.aws.sdk.bom)) + implementation(libs.aws.sdk.s3) + testImplementation(platform(libs.junit.bom)) testImplementation(libs.junit.jupiter) testRuntimeOnly(libs.junit.platform.launcher) @@ -51,6 +60,15 @@ dependencies { // is scoped to testImplementation only, not the dependency added above for production code. testImplementation(libs.josdk) + // Test-only, for FabricPushTokenRepositoryTest (phase 6): the same `@EnableKubernetesMockClient` + // fake-but-CRUD-real Kubernetes API server operator/build.gradle.kts already uses, needed here + // to prove the cluster-wide, label-selected Secret lookup actually works -- an in-memory fake + // repository (as InMemoryPushTokenRepository provides for the controller-level tests) cannot + // prove that the real fabric8 `inAnyNamespace().withLabel(...)` query and Secret.data + // base64 decoding are wired correctly. + testImplementation(libs.fabric8.junit) + testImplementation(libs.fabric8.server.mock) + // Phase 5a consolidation: both parallel worktrees reported this as missing, which meant // every existing test called controller/repository methods directly instead of going // through the real embedded server -- so role enforcement and 404-vs-403 error mapping over @@ -65,9 +83,13 @@ dependencies { // Test-only, for TenantIsolationIntegrationTest: a real k3s API server via Testcontainers, // the same pattern operator/build.gradle.kts and ingest/build.gradle.kts already use. + // MinIO backs MultipartUploadServiceIntegrationTest (phase 6): the only way to actually prove + // a presigned UploadPart URL is confined to its signed key/size is to drive real HTTP PUTs + // against a real S3-compatible server -- see that test's Javadoc. testImplementation(platform(libs.testcontainers.bom)) testImplementation(libs.testcontainers.junit) testImplementation(libs.testcontainers.k3s) + testImplementation(libs.testcontainers.minio) } // io.micronaut.test:micronaut-test-bom imports its own, newer org.testcontainers:testcontainers- diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/FabricWorldIngestRepository.java b/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/FabricWorldIngestRepository.java new file mode 100644 index 0000000..1b15e37 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/FabricWorldIngestRepository.java @@ -0,0 +1,38 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.ingest; + +import io.fabric8.kubernetes.client.KubernetesClient; +import jakarta.inject.Singleton; +import net.onelitefeather.apus.operator.api.WorldIngest; + +/** {@link WorldIngestRepository} backed by a real {@link KubernetesClient}. */ +@Singleton +public class FabricWorldIngestRepository implements WorldIngestRepository { + + private final KubernetesClient client; + + public FabricWorldIngestRepository(KubernetesClient client) { + this.client = client; + } + + @Override + public WorldIngest create(String namespace, WorldIngest ingest) { + return client.resources(WorldIngest.class).inNamespace(namespace).resource(ingest).create(); + } +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/WorldIngestRepository.java b/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/WorldIngestRepository.java new file mode 100644 index 0000000..8e5720b --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/ingest/WorldIngestRepository.java @@ -0,0 +1,35 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.ingest; + +import net.onelitefeather.apus.operator.api.WorldIngest; + +/** + * Write access to {@link WorldIngest} custom resources, always scoped to a single namespace -- + * the caller never gets to pick which one. Used by {@code + * net.onelitefeather.apus.api.rest.push.PushController} to create the {@link WorldIngest}(s) a + * {@code POST /api/push/{token}} report triggers (design spec §6.4, §11.1): a push source's + * {@code WorldIngest} is created directly by that HTTP call rather than by a poll/reconcile loop, + * exactly the "same code path" §6.4 describes pull and push sources converging on -- just + * triggered from a different place. An interface so controller tests can supply an in-memory + * fake; see {@code net.onelitefeather.apus.api.rest.tenant.TenantRepository}'s Javadoc for why. + */ +public interface WorldIngestRepository { + + WorldIngest create(String namespace, WorldIngest ingest); +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepository.java b/api/src/main/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepository.java new file mode 100644 index 0000000..542cea5 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepository.java @@ -0,0 +1,156 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.client.KubernetesClient; +import jakarta.inject.Singleton; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.util.Base64; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import net.onelitefeather.apus.operator.tenant.PushTokenSecrets; +import net.onelitefeather.apus.operator.tenant.TenantReconciler; + +/** + * {@link PushTokenRepository} backed by Kubernetes {@link Secret}s -- specifically, the ones + * {@link TenantReconciler} provisions (phase 6 task 2: one per tenant, on tenant creation). Not + * backed by any {@code WorldSourceSpec}/{@code TenantSpec} CRD field -- neither carries a token + * field -- so a token lives entirely as a plain, built-in Kubernetes resource instead, exactly + * like the S3/Pterodactyl credentials {@code WorldSourceSpec.*.credentialsSecretRef} already + * reference (design spec §10.1's "eigene S3-Credentials als Secret"). + * + *

Expected shape -- the exact contract {@link PushTokenSecrets} defines and {@link + * TenantReconciler} fulfils, re-exposed here as {@code public static final} fields so existing + * callers/tests of this class do not need to reach into {@code operator.tenant} themselves: + * + *

    + *
  • lives in the tenant's own namespace ({@code bluemap-}), like every other + * tenant-scoped resource (design spec §10.1); + *
  • labelled {@value #SERVICE_TOKEN_LABEL_KEY}: {@value #SERVICE_TOKEN_LABEL_VALUE} -- this + * is the only way this class finds it, since the namespace is exactly what a raw token does + * not carry; + *
  • {@code data.token} (or equivalently {@code stringData.token} at creation time) holds the + * raw shared-secret value the Paper plugin also holds. + *
+ * + *

Why a cluster-wide list, and the RBAC trade-off this implies. {@code POST + * /api/push/{token}} carries nothing but the token -- no tenant name, no namespace, no JWT to + * read a claim from (see {@link PushTokenRepository}'s Javadoc for why this endpoint is unlike + * every other one in this module). The token itself is the only input, so resolving it to a + * namespace necessarily means searching across namespaces; this is exactly the kind of + * cluster-wide, cross-tenant read design spec §10.3 reserves for the backend's own ServiceAccount + * ("Das Backend ist der Durchsetzungspunkt"). The label scopes that search to service-token + * Secrets specifically, not every Secret in the cluster -- but Kubernetes RBAC has no way to + * restrict a grant by a resource's label or content, only by resource type, verb and (for + * {@code get}/{@code update}/{@code patch}/{@code delete}, not {@code list}/{@code watch}) + * {@code resourceNames}. Concretely, this means: + * + *

    + *
  • the narrowest RBAC grant that actually makes {@code resolveNamespace} as implemented + * here work is a {@code ClusterRole} scoped to exactly {@code resources: ["secrets"]}, + * {@code verbs: ["get", "list"]} -- nothing else (no {@code watch}, no other resource + * types, no write verbs) -- bound to the api ServiceAccount via a {@code + * ClusterRoleBinding}. This is still, unavoidably, read access to every Secret in the + * cluster (RBAC cannot see the label filter passed in the list query), which is broader + * than the task brief's "engster Weg, der funktioniert" ("narrowest path that works") + * ideally allows -- flagged as a known trade-off, not silently accepted; + *
  • a genuinely narrower alternative exists but was deliberately not implemented here, + * to avoid an invasive rewrite of this already-tested class: since {@link + * PushTokenSecrets#SECRET_NAME} is now a fixed name, {@code resolveNamespace} could instead + * enumerate tenant namespaces (via the cluster-scoped {@code Tenant} CR, already listable + * by {@code TenantRepository} for platform-admin features) and {@code get} -- never + * {@code list} -- the fixed-name Secret in each one. That would let the RBAC grant become + * {@code resources: ["secrets"]}, {@code resourceNames: ["apus-push-token"]}, {@code verbs: + * ["get"]} -- truly scoped to only ever reading a Secret literally named {@code + * apus-push-token}, in any namespace, and nothing else. Left as a follow-up so it can be + * done with its own test coverage rather than as a side effect of wiring up token creation. + *
+ * + *

Constant-time, exhaustive comparison. {@link #resolveNamespace} runs {@link + * MessageDigest#isEqual(byte[], byte[])} -- the JDK's documented constant-time byte comparison, + * not {@code String.equals}/{@code Arrays.equals} which both short-circuit on the first + * differing byte and would leak a correct token prefix through response timing one guess at a + * time -- against *every* candidate Secret, never returning early once a match is found. Stopping + * early would leak, through timing, how many service-token Secrets exist before the caller's own + * tenant's -- a narrower signal than a whole token, but still cross-tenant information this + * endpoint must not leak (task brief: "niemals einen Hinweis darauf, ob es den Mandanten gibt"). + */ +@Singleton +public class FabricPushTokenRepository implements PushTokenRepository { + + /** See the class Javadoc's "Expected shape" for the full Secret contract this key is part of. */ + public static final String SERVICE_TOKEN_LABEL_KEY = PushTokenSecrets.LABEL_KEY; + + public static final String SERVICE_TOKEN_LABEL_VALUE = PushTokenSecrets.LABEL_VALUE; + + /** The key under {@code Secret.data} (base64-encoded, as all Secret data is) holding the raw token. */ + public static final String TOKEN_DATA_KEY = PushTokenSecrets.TOKEN_KEY; + + private final KubernetesClient client; + + public FabricPushTokenRepository(KubernetesClient client) { + this.client = client; + } + + @Override + public Optional resolveNamespace(String rawToken) { + if (rawToken == null || rawToken.isBlank()) { + return Optional.empty(); + } + byte[] supplied = rawToken.getBytes(StandardCharsets.UTF_8); + + List candidates = client.secrets() + .inAnyNamespace() + .withLabel(SERVICE_TOKEN_LABEL_KEY, SERVICE_TOKEN_LABEL_VALUE) + .list() + .getItems(); + + // Scans every candidate and never returns on the first match -- see the class Javadoc's + // "Constant-time, exhaustive comparison" for why an early return would itself be a + // (narrower, but still real) timing side-channel. + String matchedNamespace = null; + for (Secret candidate : candidates) { + byte[] stored = decodedToken(candidate); + boolean matches = stored != null && MessageDigest.isEqual(stored, supplied); + if (matches) { + matchedNamespace = candidate.getMetadata().getNamespace(); + } + } + return Optional.ofNullable(matchedNamespace); + } + + private static byte[] decodedToken(Secret secret) { + Map data = secret.getData(); + if (data == null) { + return null; + } + String encoded = data.get(TOKEN_DATA_KEY); + if (encoded == null) { + return null; + } + try { + return Base64.getDecoder().decode(encoded); + } catch (IllegalArgumentException e) { + // A malformed Secret must not crash the whole lookup for every other tenant's token. + return null; + } + } +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushController.java b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushController.java new file mode 100644 index 0000000..a82d136 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushController.java @@ -0,0 +1,143 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import io.micronaut.core.annotation.Nullable; +import io.micronaut.http.HttpResponse; +import io.micronaut.http.annotation.Body; +import io.micronaut.http.annotation.Controller; +import io.micronaut.http.annotation.PathVariable; +import io.micronaut.http.annotation.Post; +import io.micronaut.security.annotation.Secured; +import io.micronaut.security.rules.SecurityRule; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import net.onelitefeather.apus.api.rest.ingest.WorldIngestRepository; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import net.onelitefeather.apus.api.rest.support.NotFoundException; +import net.onelitefeather.apus.api.rest.worldsource.WorldSourceRepository; +import net.onelitefeather.apus.operator.api.WorldIngest; +import net.onelitefeather.apus.operator.api.WorldSource; + +/** + * {@code POST /api/push/{token}} -- the completion report a {@code push}-type {@code + * WorldSource}'s owner (the Paper server plugin) sends once it has finished writing world data + * directly into its tenant's staging prefix in S3. Creates one {@link WorldIngest} per world the + * target source has configured (design spec §8.3, mirroring {@code + * WorldSourceReconciler#triggerIngests}'s per-world loop for pull sources -- §6.4's "beide Wege + * münden in denselben Code-Pfad"). + * + *

The one endpoint in this module that is not JWT-authenticated. {@code + * @Secured(SecurityRule.IS_ANONYMOUS)} is deliberate, not an oversight: this call carries a + * service token in the path, never a bearer JWT (see {@link PushTokenRepository}'s Javadoc for + * why), so Micronaut Security's JWT filter has nothing to validate here -- authentication is this + * controller's own job, done entirely by {@link PushTokenRepository#resolveNamespace}. + * + *

The namespace always comes from the token, never from the request body (task brief's + * central rule for this endpoint, mirroring {@code TenantResolver}'s for JWT-authenticated ones). + * {@code request.sourceName()} only selects *which* of the token's own tenant's sources to target + * -- {@link WorldSourceRepository#find} is already scoped to that token-resolved namespace before + * {@code sourceName} is ever looked at, so a source name that happens to exist in a different + * tenant cannot be reached this way. + * + *

Every failure path is a plain 404 or 400, uniformly. An unknown/wrong-tenant token, an + * unknown source name, and a source that is not of type {@code push} all produce {@link + * NotFoundException} -- never a distinct status or message that would let a caller tell "this + * token is wrong" apart from "this token is right but that source doesn't exist" apart from "that + * tenant doesn't exist at all". A malformed request body (missing {@code sourceName}/{@code + * version}, or a source with no worlds configured) is the caller's own request being invalid, + * which is safe to report distinctly (400) -- it happens only *after* the token has already + * proven the caller belongs to a real tenant, so it discloses nothing about any other one. + */ +@Controller("/api/push") +@Secured(SecurityRule.IS_ANONYMOUS) +public class PushController { + + private static final String TYPE_PUSH = "push"; + + private final PushTokenRepository tokenRepository; + private final WorldSourceRepository sourceRepository; + private final WorldIngestRepository ingestRepository; + + public PushController( + PushTokenRepository tokenRepository, + WorldSourceRepository sourceRepository, + WorldIngestRepository ingestRepository) { + this.tokenRepository = tokenRepository; + this.sourceRepository = sourceRepository; + this.ingestRepository = ingestRepository; + } + + @Post("/{token}") + public HttpResponse report( + @PathVariable String token, @Nullable @Body PushReportRequest request) { + // Resolved before the request body is even inspected: an invalid token must fail + // identically regardless of what (if anything) the body contains. + String namespace = tokenRepository + .resolveNamespace(token) + .orElseThrow(() -> new NotFoundException("no push source authorized for this token")); + + if (request == null || isBlank(request.sourceName()) || isBlank(request.version())) { + throw new BadRequestException("sourceName and version are both required"); + } + + WorldSource source = sourceRepository + .find(namespace, request.sourceName()) + .filter(s -> TYPE_PUSH.equals(s.getSpec().getType())) + .orElseThrow(() -> new NotFoundException( + "no push source '" + request.sourceName() + "' in namespace '" + namespace + "'")); + + List worlds = source.getSpec().getWorlds(); + if (worlds.isEmpty()) { + throw new BadRequestException("source '" + request.sourceName() + "' has no configured worlds"); + } + + List created = new ArrayList<>(); + for (WorldSource.WorldSelector selector : worlds) { + WorldIngest ingest = new WorldIngest(); + ingest.getMetadata() + .setGenerateName( + sanitize(request.sourceName()) + "-" + sanitize(selector.getName()) + "-push-"); + ingest.getSpec().getSourceRef().setName(request.sourceName()); + ingest.getSpec().setSourceVersion(request.version()); + ingest.getSpec().setWorldName(selector.getName()); + + WorldIngest result = ingestRepository.create(namespace, ingest); + created.add(result.getMetadata().getName()); + } + + return HttpResponse.created(new PushReportResponse(created)); + } + + private static boolean isBlank(String value) { + return value == null || value.isBlank(); + } + + /** + * Kubernetes {@code generateName} prefixes must be valid RFC 1123 DNS subdomain label + * fragments (lowercase alphanumeric and {@code -}); neither a {@code WorldSource} name nor, + * especially, a Minecraft world name (e.g. {@code world_nether}) is guaranteed to already be + * one. + */ + private static String sanitize(String value) { + String lower = value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9-]", "-"); + String trimmed = lower.replaceAll("^-+", "").replaceAll("-+$", ""); + return trimmed.isEmpty() ? "x" : trimmed; + } +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportRequest.java b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportRequest.java new file mode 100644 index 0000000..0ac9576 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportRequest.java @@ -0,0 +1,34 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import io.micronaut.serde.annotation.Serdeable; + +/** + * Request body for {@code POST /api/push/{token}}. Deliberately carries no tenant/namespace field + * -- that comes only from the token (see {@code PushTokenRepository}'s Javadoc). {@code + * sourceName} names one of the token's own tenant's {@code WorldSource} resources (of type {@code + * push}) -- a tenant may run more than one Paper server/world, each pushing to a different + * source, so the token alone (tenant-bound, not source-bound, design spec §10.3) is not enough to + * pick one. {@code version} is the identifier the plugin used as the staged object's key suffix + * when it wrote the data directly to S3 -- becomes {@code WorldIngest.spec.sourceVersion}, and + * from there the {@code SourceVersion.id()} the ingest job's {@code PushSourceConnector} (module + * {@code ingest}) fetches by. + */ +@Serdeable +public record PushReportRequest(String sourceName, String version) {} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportResponse.java b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportResponse.java new file mode 100644 index 0000000..91a0e28 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushReportResponse.java @@ -0,0 +1,25 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import io.micronaut.serde.annotation.Serdeable; +import java.util.List; + +/** {@code POST /api/push/{token}}'s response -- the names of the {@code WorldIngest} resources it created, one per configured world on the target source. */ +@Serdeable +public record PushReportResponse(List worldIngests) {} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushTokenRepository.java b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushTokenRepository.java new file mode 100644 index 0000000..17a914a --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/push/PushTokenRepository.java @@ -0,0 +1,53 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import java.util.Optional; + +/** + * Resolves a raw {@code world:push} service token (design spec §10.3) to the single namespace it + * authorizes -- the only lookup {@code PushController} is allowed to make before it knows which + * tenant a {@code POST /api/push/{token}} call belongs to. + * + *

This is deliberately not a JWT. Every other authenticated endpoint in this module + * validates a JWT issued by the identity broker (design spec §10.3, {@code + * PrincipalResolver}/{@code TenantResolver}). A push token is different on purpose: it is a + * long-lived, tenant-bound bearer secret a Paper server plugin holds, deliberately not tied to + * any user login (§10.3: "sonst würde das Ausscheiden einer Person den Server-Upload + * lahmlegen"). It arrives as a path segment, not a bearer JWT, and {@link #resolveNamespace} must + * compare it against every known token in constant time (see {@code + * FabricPushTokenRepository#resolveNamespace} for how) so that no amount of failed guesses lets + * an attacker learn a correct token one character at a time, and a non-matching token must look + * identical -- in both response and timing -- whether or not the tenant it might have belonged to + * even exists. + * + *

An interface so controller tests can supply an in-memory fake; see {@code + * net.onelitefeather.apus.api.rest.tenant.TenantRepository}'s Javadoc for why. + */ +public interface PushTokenRepository { + + /** + * @param rawToken the token exactly as it arrived in the {@code POST /api/push/{token}} path + * segment; {@code null} or blank is a valid input and always resolves to {@link + * Optional#empty()} + * @return the namespace this token authorizes push access to, or {@link Optional#empty()} if + * no known token matches -- never distinguishes "no such token" from "token valid for a + * different tenant" in what it returns + */ + Optional resolveNamespace(String rawToken); +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadRequest.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadRequest.java new file mode 100644 index 0000000..9a530ce --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadRequest.java @@ -0,0 +1,31 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.serde.annotation.Serdeable; + +/** + * Request body for {@code POST /api/uploads/{uploadId}/complete} -- {@code sourceName}, {@code + * version} and {@code fileName} must match what {@code POST /api/uploads} originally returned + * (they are what {@link MultipartUploadService#stagingKey} recomputes the object key from); no + * part list is required, since {@link MultipartUploadService#completeUpload} reads the + * authoritative part sizes/ETags back from S3 itself via {@code ListParts} rather than trusting + * anything the caller claims about what it uploaded. + */ +@Serdeable +public record CompleteUploadRequest(String sourceName, String version, String fileName) {} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadResponse.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadResponse.java new file mode 100644 index 0000000..e7224ef --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CompleteUploadResponse.java @@ -0,0 +1,24 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.serde.annotation.Serdeable; + +/** {@code POST /api/uploads/{uploadId}/complete}'s response -- the finished object's key and its real, S3-verified size. */ +@Serdeable +public record CompleteUploadResponse(String key, String version, long totalBytes, int partCount) {} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadRequest.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadRequest.java new file mode 100644 index 0000000..26f355d --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadRequest.java @@ -0,0 +1,29 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.serde.annotation.Serdeable; + +/** + * Request body for {@code POST /api/uploads}. {@code sourceName} names one of the caller's own + * tenant's {@code WorldSource} resources (of type {@code upload}) -- resolved against the + * caller's namespace by the controller, exactly like every other write in this module (design + * spec §10.3); this request carries no namespace/tenant field of its own. + */ +@Serdeable +public record CreateUploadRequest(String sourceName, String fileName, long sizeBytes) {} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadResponse.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadResponse.java new file mode 100644 index 0000000..5abf091 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/CreateUploadResponse.java @@ -0,0 +1,44 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.serde.annotation.Serdeable; +import java.time.Instant; +import java.util.List; + +/** + * {@code POST /api/uploads}'s response: everything the caller needs to upload every part + * directly to S3 and then call {@code POST /api/uploads/{uploadId}/complete}. Carries no + * credentials -- each {@link PresignedPart#url()} already has its own signature embedded; that is + * the entire point of a presigned URL. + */ +@Serdeable +public record CreateUploadResponse( + String uploadId, + String bucket, + String key, + String version, + String fileName, + List parts, + long partSizeBytes, + Instant expiresAt) { + + /** One presigned {@code UploadPart} slot -- the caller {@code PUT}s that part's bytes directly to {@code url}. */ + @Serdeable + public record PresignedPart(int partNumber, long sizeBytes, String url) {} +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadService.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadService.java new file mode 100644 index 0000000..a267992 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadService.java @@ -0,0 +1,309 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.context.annotation.Value; +import jakarta.inject.Singleton; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import java.util.regex.Pattern; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload; +import software.amazon.awssdk.services.s3.model.CompletedPart; +import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; +import software.amazon.awssdk.services.s3.model.ListPartsRequest; +import software.amazon.awssdk.services.s3.model.ListPartsResponse; +import software.amazon.awssdk.services.s3.model.Part; +import software.amazon.awssdk.services.s3.model.UploadPartRequest; +import software.amazon.awssdk.services.s3.presigner.S3Presigner; +import software.amazon.awssdk.services.s3.presigner.model.PresignedUploadPartRequest; +import software.amazon.awssdk.services.s3.presigner.model.UploadPartPresignRequest; + +/** + * Presigned multipart uploads into the staging bucket (design spec §11.1) -- the actual bulk + * data transfer for {@code POST /api/uploads} happens directly between the caller and S3, never + * through this API. This class only ever hands out presigned URLs for {@code UploadPart}; {@code + * CreateMultipartUpload}, {@code ListParts}, {@code CompleteMultipartUpload} and {@code + * AbortMultipartUpload} are all performed here, directly, with this backend's own staging + * credentials -- never presigned for a caller to invoke themselves. + * + *

Why completion is not presigned too. {@code S3Presigner} can presign a {@code + * CompleteMultipartUploadRequest} just as well as an {@code UploadPartRequest}. Handing that out + * as well would mean this backend never sees the completion call at all -- and completion is the + * one point in the whole multipart lifecycle where the *actual* total bytes uploaded are known + * (via {@code ListParts}, which reads real, S3-recorded part sizes, not anything the client + * claims). Keeping completion as an authenticated call this backend performs itself is what makes + * {@link #completeUpload}'s size check a real enforcement point rather than a suggestion: a + * caller who oversubscribes their declared size and then still uploads a huge part earns an + * {@link AbortMultipartUploadRequest}, not a usable object. + * + *

Prefix confinement is structural, not advisory. {@link #stagingKey} is the only place + * an S3 key is ever built, and it is a pure function of a server-derived {@code namespace} (from + * {@code TenantResolver}, itself from the caller's validated JWT -- never client input) plus a + * {@code sourceName} the controller has already confirmed is a real {@code WorldSource} in that + * same namespace. Whatever a caller passes as {@code version}/{@code fileName} therefore can only + * ever select an object *within* {@code ///...} -- S3 keys have no + * {@code ..}-style traversal semantics, so there is no sequence of characters in either field that + * escapes that subtree into a different tenant's. + * + *

What is, and is not, actually enforced on size -- read before trusting this class fully. + * See the phase 6 task report for the full analysis; the short version: + * + *

    + *
  • Enforced, verified: {@link #completeUpload} sums the real, S3-recorded sizes of + * every part via {@code ListParts} and aborts (never completes) an upload whose actual + * total exceeds {@link #maxUploadBytes}. An oversized upload therefore never becomes a + * retrievable object, regardless of what any individual presigned part URL allowed through. + *
  • Enforced, empirically confirmed against real MinIO (2026-08-09, {@code + * MultipartUploadServiceIntegrationTest#aPartExceedingItsPresignedSizeIsRejectedBeforeItIsAccepted}): + * {@link #createUpload} sets {@code Content-Length} on each presigned {@code UploadPart} + * request to the exact byte count that part was sized for. The AWS SDK v2 presigner + * includes {@code Content-Length} among that URL's signed headers, so a client sending more + * bytes than declared for a given part is rejected with HTTP 403 {@code + * SignatureDoesNotMatch} before those extra bytes are accepted -- confirmed by driving a + * real oversized {@code PUT} against a real MinIO instance, not inferred from SDK + * documentation. Not independently re-verified against Ceph RGW (the production backend + * design spec §9.1 names) -- both implement SigV4 presigned-URL validation the same way, but + * that specific claim is an inference from the MinIO result, not a second measurement. + *
  • Not this class's job at all, and does not need to be: the tenant's actual storage + * budget. Design spec §10.2 is explicit that {@code Tenant.spec.storage.quota} is enforced + * by Ceph RGW itself, independent of anything the application does or gets wrong -- + * {@link #maxUploadBytes} exists only to bound a single upload's blast radius (an absurd + * one-shot request), not to replace that quota. + *
+ */ +@Singleton +public class MultipartUploadService { + + /** Hard defensive ceiling on issued part URLs, well under S3's own 10 000-part protocol limit. */ + private static final int MAX_PARTS = 2000; + + private static final Pattern SAFE_FILE_NAME = Pattern.compile("[A-Za-z0-9._-]{1,255}"); + private static final Pattern SAFE_VERSION = Pattern.compile("[A-Za-z0-9-]{1,128}"); + + private final S3Client s3Client; + private final S3Presigner presigner; + private final String bucket; + private final String prefix; + private final long partSizeBytes; + private final long maxUploadBytes; + private final Duration urlExpiry; + + public MultipartUploadService( + S3Client s3Client, + S3Presigner presigner, + @Value("${apus.staging.bucket}") String bucket, + @Value("${apus.staging.prefix:staging/}") String prefix, + @Value("${apus.staging.part-size-bytes:67108864}") long partSizeBytes, + @Value("${apus.staging.max-upload-bytes:10737418240}") long maxUploadBytes, + @Value("${apus.staging.url-expiry-seconds:900}") long urlExpirySeconds) { + this.s3Client = s3Client; + this.presigner = presigner; + this.bucket = bucket; + this.prefix = normalisePrefix(prefix); + this.partSizeBytes = partSizeBytes; + this.maxUploadBytes = maxUploadBytes; + this.urlExpiry = Duration.ofSeconds(urlExpirySeconds); + } + + /** + * Initiates a multipart upload and presigns every part slot it will need for {@code + * declaredSizeBytes}. + * + * @param namespace the caller's own namespace, from {@code TenantResolver} -- never anything + * else, see the class Javadoc + * @param sourceName the target {@code WorldSource}'s name, already confirmed by the caller to + * exist (as type {@code upload}) in {@code namespace} + * @param fileName the object's file name (e.g. {@code "world.tar.gz"}), kept as the staged + * key's final segment so its extension survives for {@code Archives.isArchive} at ingest + * time; validated against {@link #SAFE_FILE_NAME} + * @param declaredSizeBytes the caller's declared total size -- used only to size/count parts + * and for the cheap upfront rejection in this method; the real check is {@link + * #completeUpload}'s + * @throws BadRequestException if {@code fileName}/{@code declaredSizeBytes} are invalid, or + * the declared size would require issuing more than {@link #MAX_PARTS} presigned URLs + */ + public CreateUploadResponse createUpload(String namespace, String sourceName, String fileName, long declaredSizeBytes) { + requireSafe(fileName, SAFE_FILE_NAME, "fileName"); + if (declaredSizeBytes <= 0) { + throw new BadRequestException("sizeBytes must be greater than zero"); + } + if (declaredSizeBytes > maxUploadBytes) { + throw new BadRequestException("sizeBytes exceeds the maximum allowed upload size of " + maxUploadBytes + " bytes"); + } + + String version = UUID.randomUUID().toString(); + String key = stagingKey(prefix, namespace, sourceName, version, fileName); + + CreateMultipartUploadResponse created = s3Client.createMultipartUpload( + CreateMultipartUploadRequest.builder().bucket(bucket).key(key).build()); + String uploadId = created.uploadId(); + + int partCount = (int) Math.ceil((double) declaredSizeBytes / (double) partSizeBytes); + if (partCount > MAX_PARTS) { + // Nothing was uploaded yet -- abort immediately rather than leaving an empty + // multipart upload dangling in S3 for no reason. + s3Client.abortMultipartUpload(AbortMultipartUploadRequest.builder() + .bucket(bucket) + .key(key) + .uploadId(uploadId) + .build()); + throw new BadRequestException("sizeBytes would require more than " + MAX_PARTS + " parts"); + } + + List parts = new ArrayList<>(partCount); + for (int partNumber = 1; partNumber <= partCount; partNumber++) { + long thisPartSize = + (partNumber < partCount) ? partSizeBytes : declaredSizeBytes - partSizeBytes * (long) (partCount - 1); + parts.add(presignPart(key, uploadId, partNumber, thisPartSize)); + } + + return new CreateUploadResponse( + uploadId, bucket, key, version, fileName, parts, partSizeBytes, Instant.now().plus(urlExpiry)); + } + + private CreateUploadResponse.PresignedPart presignPart(String key, String uploadId, int partNumber, long partSize) { + UploadPartRequest partRequest = UploadPartRequest.builder() + .bucket(bucket) + .key(key) + .uploadId(uploadId) + .partNumber(partNumber) + // Pins this part's exact size into the presigned URL's signature -- empirically + // confirmed (see the class Javadoc's "What is, and is not, actually enforced" + // section) to make a larger upload fail with 403 SignatureDoesNotMatch. + .contentLength(partSize) + .build(); + UploadPartPresignRequest presignRequest = UploadPartPresignRequest.builder() + .signatureDuration(urlExpiry) + .uploadPartRequest(partRequest) + .build(); + PresignedUploadPartRequest presigned = presigner.presignUploadPart(presignRequest); + return new CreateUploadResponse.PresignedPart(partNumber, partSize, presigned.url().toString()); + } + + /** + * Finalises a multipart upload -- the real size enforcement point, see the class Javadoc. + * Sums the actual, S3-recorded size of every uploaded part via {@code ListParts} and only + * calls {@code CompleteMultipartUpload} if that real total is within {@link + * #maxUploadBytes}; otherwise aborts the upload and rejects the request. The key is + * recomputed from {@code namespace}/{@code sourceName}/{@code version}/{@code fileName} + * exactly as {@link #createUpload} built it -- never taken from the caller directly -- so + * completion is confined to the same namespace-scoped prefix creation was. + * + * @throws BadRequestException if no parts were uploaded, the real total exceeds {@link + * #maxUploadBytes} (the upload is aborted before this is thrown), or {@code version}/ + * {@code fileName} fail their format checks + */ + public CompleteUploadResponse completeUpload( + String namespace, String sourceName, String version, String fileName, String uploadId) { + requireSafe(fileName, SAFE_FILE_NAME, "fileName"); + requireSafe(version, SAFE_VERSION, "version"); + if (uploadId == null || uploadId.isBlank()) { + throw new BadRequestException("uploadId must not be blank"); + } + + String key = stagingKey(prefix, namespace, sourceName, version, fileName); + + List allParts = listAllParts(key, uploadId); + long totalBytes = allParts.stream().mapToLong(Part::size).sum(); + + if (allParts.isEmpty()) { + throw new BadRequestException("no parts were uploaded for this upload"); + } + if (totalBytes > maxUploadBytes) { + s3Client.abortMultipartUpload(AbortMultipartUploadRequest.builder() + .bucket(bucket) + .key(key) + .uploadId(uploadId) + .build()); + throw new BadRequestException( + "upload's actual total size (" + totalBytes + " bytes) exceeds the maximum allowed (" + + maxUploadBytes + " bytes); the upload was aborted"); + } + + List completedParts = allParts.stream() + .map(part -> CompletedPart.builder() + .partNumber(part.partNumber()) + .eTag(part.eTag()) + .build()) + .toList(); + + s3Client.completeMultipartUpload(CompleteMultipartUploadRequest.builder() + .bucket(bucket) + .key(key) + .uploadId(uploadId) + .multipartUpload(CompletedMultipartUpload.builder().parts(completedParts).build()) + .build()); + + return new CompleteUploadResponse(key, version, totalBytes, allParts.size()); + } + + /** Pages through every uploaded part -- authoritative, S3-recorded sizes/ETags, never trusting anything the caller claims. */ + private List listAllParts(String key, String uploadId) { + List all = new ArrayList<>(); + Integer partNumberMarker = null; + boolean truncated = true; + while (truncated) { + var requestBuilder = ListPartsRequest.builder().bucket(bucket).key(key).uploadId(uploadId); + if (partNumberMarker != null) { + requestBuilder.partNumberMarker(partNumberMarker); + } + ListPartsResponse response = s3Client.listParts(requestBuilder.build()); + all.addAll(response.parts()); + truncated = Boolean.TRUE.equals(response.isTruncated()); + if (truncated) { + partNumberMarker = response.nextPartNumberMarker(); + if (partNumberMarker == null) { + break; + } + } + } + return all; + } + + /** + * The single place a staging S3 key is ever constructed -- see the class Javadoc's "Prefix + * confinement is structural, not advisory". Package-private and {@code static} so it can be + * unit-tested directly, with no S3 client/credentials/Micronaut context involved, proving the + * confinement property for arbitrary (including adversarial) {@code sourceName}/{@code + * version}/{@code fileName} inputs. + */ + static String stagingKey(String prefix, String namespace, String sourceName, String version, String fileName) { + return normalisePrefix(prefix) + namespace + "/" + sourceName + "/" + version + "/" + fileName; + } + + private static String normalisePrefix(String prefix) { + if (prefix == null || prefix.isBlank()) { + return ""; + } + return prefix.endsWith("/") ? prefix : prefix + "/"; + } + + private static void requireSafe(String value, Pattern pattern, String fieldName) { + if (value == null || !pattern.matcher(value).matches()) { + throw new BadRequestException(fieldName + " is missing or contains characters outside " + pattern.pattern()); + } + } +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/StagingS3ClientFactory.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/StagingS3ClientFactory.java new file mode 100644 index 0000000..cd64a1b --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/StagingS3ClientFactory.java @@ -0,0 +1,103 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.context.annotation.Factory; +import io.micronaut.context.annotation.Value; +import jakarta.inject.Singleton; +import java.net.URI; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3Configuration; +import software.amazon.awssdk.services.s3.presigner.S3Presigner; + +/** + * The {@link S3Client}/{@link S3Presigner} pair {@link MultipartUploadService} uses against the + * platform-wide staging bucket -- one shared bucket, tenant isolation coming entirely from the + * key prefix each presigned URL is scoped to (see that class's Javadoc), the same "one bucket, + * many mandant-scoped prefixes" reading of design spec §11.1 the task brief's "auf das + * Staging-Präfix genau dieses Mandanten" language implies. Credentials are therefore + * platform-wide, not per-tenant -- mirroring how design spec §10.2 already describes the RGW + * admin-ops credentials used for quota polling ("Die Zugangsdaten dafür sind plattformweit, nicht + * mandantengebunden"). + * + *

{@code @Value} injection directly into {@code @Factory} methods, not a {@code + * @ConfigurationProperties} class -- matches this module's existing convention (see {@code + * net.onelitefeather.apus.api.events.LogSourceFactory}). No property here has a hardcoded + * default that would silently point at a real bucket; every required one fails Micronaut startup + * with a clear "missing configuration" error if unset, the same fail-fast posture {@code + * application.yml}'s JWT properties already have. + */ +@Factory +class StagingS3ClientFactory { + + private static final String DEFAULT_REGION = "us-east-1"; + + @Singleton + S3Client stagingS3Client( + @Value("${apus.staging.endpoint:}") String endpoint, + @Value("${apus.staging.region:" + DEFAULT_REGION + "}") String region, + @Value("${apus.staging.access-key-id:}") String accessKeyId, + @Value("${apus.staging.secret-access-key:}") String secretAccessKey) { + var builder = S3Client.builder().region(Region.of(region)).credentialsProvider(credentials(accessKeyId, secretAccessKey)); + if (endpoint != null && !endpoint.isBlank()) { + // S3-compatible stores (Rook/Ceph, MinIO, ...) need an endpoint override and + // path-style bucket addressing -- see S3SourceConnector.buildClient (ingest module) + // for the same reasoning applied to a pull source's own client. + builder = builder.endpointOverride(URI.create(endpoint)).forcePathStyle(true); + } + return builder.build(); + } + + @Singleton + S3Presigner stagingS3Presigner( + @Value("${apus.staging.endpoint:}") String endpoint, + @Value("${apus.staging.region:" + DEFAULT_REGION + "}") String region, + @Value("${apus.staging.access-key-id:}") String accessKeyId, + @Value("${apus.staging.secret-access-key:}") String secretAccessKey) { + var builder = + S3Presigner.builder().region(Region.of(region)).credentialsProvider(credentials(accessKeyId, secretAccessKey)); + if (endpoint != null && !endpoint.isBlank()) { + builder = builder.endpointOverride(URI.create(endpoint)) + // Path-style addressing must be requested separately from the presigner's own + // service configuration -- endpointOverride() alone does not imply it, and a + // presigned URL built for virtual-hosted-style addressing against a + // non-AWS-DNS endpoint would simply not resolve for whoever tries to use it. + // Checksum validation is disabled for the same reason S3-compatible stores + // generally need it off for presigned uploads: the SDK would otherwise try to + // add a checksum trailer/header the presigned request was never signed to + // include, breaking the signature for whoever performs the actual PUT. + .serviceConfiguration(S3Configuration.builder() + .pathStyleAccessEnabled(true) + .checksumValidationEnabled(false) + .build()); + } + return builder.build(); + } + + private static AwsCredentialsProvider credentials(String accessKeyId, String secretAccessKey) { + if (accessKeyId != null && !accessKeyId.isBlank() && secretAccessKey != null && !secretAccessKey.isBlank()) { + return StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKeyId, secretAccessKey)); + } + return DefaultCredentialsProvider.builder().build(); + } +} diff --git a/api/src/main/java/net/onelitefeather/apus/api/rest/upload/UploadController.java b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/UploadController.java new file mode 100644 index 0000000..84632b0 --- /dev/null +++ b/api/src/main/java/net/onelitefeather/apus/api/rest/upload/UploadController.java @@ -0,0 +1,126 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import io.micronaut.http.HttpResponse; +import io.micronaut.http.annotation.Body; +import io.micronaut.http.annotation.Controller; +import io.micronaut.http.annotation.PathVariable; +import io.micronaut.http.annotation.Post; +import io.micronaut.security.annotation.Secured; +import io.micronaut.security.authentication.Authentication; +import io.micronaut.security.rules.SecurityRule; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import net.onelitefeather.apus.api.rest.support.NotFoundException; +import net.onelitefeather.apus.api.rest.worldsource.WorldSourceRepository; +import net.onelitefeather.apus.api.security.ApusPrincipal; +import net.onelitefeather.apus.api.security.ForbiddenException; +import net.onelitefeather.apus.api.security.TenantResolver; +import net.onelitefeather.apus.api.support.PrincipalResolver; + +/** + * {@code POST /api/uploads} and {@code POST /api/uploads/{uploadId}/complete} -- the caller's own + * tenant only (design spec §10.3, §11.1), exactly like every other JWT-authenticated controller in + * this module. The namespace always comes from {@link TenantResolver}, never from the request + * body; {@link MultipartUploadService} then derives the S3 key from that namespace alone, so a + * caller can never reach outside their own tenant's staging prefix regardless of what {@code + * sourceName}/{@code version}/{@code fileName} they supply (see that class's Javadoc). + * + *

Requires {@link ApusPrincipal#canWrite()} for both operations -- initiating and completing an + * upload are both writes, exactly like {@code POST /api/sources} and {@code POST + * /api/maps/{id}/render}. + */ +@Controller("/api/uploads") +@Secured(SecurityRule.IS_AUTHENTICATED) +public class UploadController { + + private static final String TYPE_UPLOAD = "upload"; + + private final WorldSourceRepository sourceRepository; + private final MultipartUploadService uploadService; + private final PrincipalResolver principalResolver; + private final TenantResolver tenantResolver; + + public UploadController( + WorldSourceRepository sourceRepository, + MultipartUploadService uploadService, + PrincipalResolver principalResolver, + TenantResolver tenantResolver) { + this.sourceRepository = sourceRepository; + this.uploadService = uploadService; + this.principalResolver = principalResolver; + this.tenantResolver = tenantResolver; + } + + @Post + public HttpResponse create(Authentication authentication, @Body CreateUploadRequest request) { + ApusPrincipal principal = principalResolver.resolve(authentication); + requireWrite(principal); + String namespace = tenantResolver.namespaceFor(principal); + + if (request == null || isBlank(request.sourceName())) { + throw new BadRequestException("sourceName is required"); + } + findOwnUploadSource(namespace, request.sourceName()); + + CreateUploadResponse response = + uploadService.createUpload(namespace, request.sourceName(), request.fileName(), request.sizeBytes()); + return HttpResponse.created(response); + } + + @Post("/{uploadId}/complete") + public HttpResponse complete( + Authentication authentication, @PathVariable String uploadId, @Body CompleteUploadRequest request) { + ApusPrincipal principal = principalResolver.resolve(authentication); + requireWrite(principal); + String namespace = tenantResolver.namespaceFor(principal); + + if (request == null || isBlank(request.sourceName())) { + throw new BadRequestException("sourceName is required"); + } + findOwnUploadSource(namespace, request.sourceName()); + + CompleteUploadResponse response = uploadService.completeUpload( + namespace, request.sourceName(), request.version(), request.fileName(), uploadId); + return HttpResponse.ok(response); + } + + /** + * Confirmed to exist, as an {@code upload}-type source, in the caller's own namespace -- + * exactly like {@code BlueMapMapController.findOwnMap} does before creating a {@code + * BlueMapRender} referencing it, and for the same reason: a foreign tenant's source name + * must fail exactly like a non-existent one (404), never leaking that it exists elsewhere. + */ + private void findOwnUploadSource(String namespace, String sourceName) { + sourceRepository + .find(namespace, sourceName) + .filter(s -> TYPE_UPLOAD.equals(s.getSpec().getType())) + .orElseThrow(() -> + new NotFoundException("no upload source '" + sourceName + "' in namespace '" + namespace + "'")); + } + + private static boolean isBlank(String value) { + return value == null || value.isBlank(); + } + + private void requireWrite(ApusPrincipal principal) { + if (!principal.canWrite()) { + throw new ForbiddenException("principal '" + principal.subject() + "' is not tenant-owner/tenant-operator"); + } + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/ingest/InMemoryWorldIngestRepository.java b/api/src/test/java/net/onelitefeather/apus/api/rest/ingest/InMemoryWorldIngestRepository.java new file mode 100644 index 0000000..4f3bd05 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/ingest/InMemoryWorldIngestRepository.java @@ -0,0 +1,55 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.ingest; + +import java.util.ArrayList; +import java.util.List; +import net.onelitefeather.apus.operator.api.WorldIngest; + +/** + * An in-memory, namespace-partitioned {@link WorldIngestRepository} fake -- emulates Kubernetes' + * {@code generateName} behaviour (a unique {@code name} is assigned on {@link #create} whenever + * only {@code generateName} was set) closely enough for {@code PushControllerTest} to assert on + * the created resources without a real or mocked cluster. Public (unlike most of this module's + * in-memory fakes) because {@code net.onelitefeather.apus.api.rest.push.PushControllerTest} is in + * a different package and needs it too. + */ +public final class InMemoryWorldIngestRepository implements WorldIngestRepository { + + private final List items = new ArrayList<>(); + private int counter = 0; + + @Override + public WorldIngest create(String namespace, WorldIngest ingest) { + if (ingest.getMetadata().getName() == null) { + String generateName = ingest.getMetadata().getGenerateName(); + ingest.getMetadata().setName((generateName == null ? "ingest-" : generateName) + (counter++)); + } + items.add(new Namespaced(namespace, ingest)); + return ingest; + } + + public List forNamespace(String namespace) { + return items.stream() + .filter(item -> item.namespace().equals(namespace)) + .map(Namespaced::resource) + .toList(); + } + + private record Namespaced(String namespace, WorldIngest resource) {} +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepositoryTest.java b/api/src/test/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepositoryTest.java new file mode 100644 index 0000000..b578f4b --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/push/FabricPushTokenRepositoryTest.java @@ -0,0 +1,112 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.fabric8.kubernetes.api.model.ObjectMetaBuilder; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import java.nio.charset.StandardCharsets; +import java.util.Base64; +import java.util.Map; +import org.junit.jupiter.api.Test; + +/** + * Proves the real fabric8 wiring {@code PushControllerTest}'s in-memory fake cannot: the + * cluster-wide, label-scoped Secret query and the base64 {@code data.token} decoding. + */ +@EnableKubernetesMockClient(crud = true) +class FabricPushTokenRepositoryTest { + + KubernetesClient client; + + private FabricPushTokenRepository repository() { + return new FabricPushTokenRepository(client); + } + + private void serviceTokenSecret(String namespace, String name, String rawToken) { + Secret secret = new SecretBuilder() + .withMetadata(new ObjectMetaBuilder() + .withName(name) + .withNamespace(namespace) + .withLabels(Map.of( + FabricPushTokenRepository.SERVICE_TOKEN_LABEL_KEY, + FabricPushTokenRepository.SERVICE_TOKEN_LABEL_VALUE)) + .build()) + .withData(Map.of( + FabricPushTokenRepository.TOKEN_DATA_KEY, + Base64.getEncoder().encodeToString(rawToken.getBytes(StandardCharsets.UTF_8)))) + .build(); + client.secrets().inNamespace(namespace).resource(secret).create(); + } + + @Test + void resolvesTheNamespaceOfTheMatchingSecret() { + serviceTokenSecret("bluemap-acme", "apus-push-token", "acme-super-secret-token"); + + var result = repository().resolveNamespace("acme-super-secret-token"); + + assertTrue(result.isPresent()); + assertEquals("bluemap-acme", result.get()); + } + + @Test + void aTokenThatMatchesNoSecretResolvesToEmpty() { + serviceTokenSecret("bluemap-acme", "apus-push-token", "acme-super-secret-token"); + + assertTrue(repository().resolveNamespace("some-other-token").isEmpty()); + } + + @Test + void blankOrNullTokenResolvesToEmptyWithoutQueryingTheCluster() { + assertTrue(repository().resolveNamespace("").isEmpty()); + assertTrue(repository().resolveNamespace(null).isEmpty()); + } + + @Test + void picksTheCorrectNamespaceAmongMultipleTenantsTokens() { + serviceTokenSecret("bluemap-acme", "apus-push-token", "acme-token"); + serviceTokenSecret("bluemap-globex", "apus-push-token", "globex-token"); + serviceTokenSecret("bluemap-initech", "apus-push-token", "initech-token"); + + assertEquals("bluemap-globex", repository().resolveNamespace("globex-token").orElseThrow()); + assertEquals("bluemap-acme", repository().resolveNamespace("acme-token").orElseThrow()); + } + + @Test + void aSecretWithoutTheServiceTokenLabelIsIgnored() { + // A Secret that merely happens to have a "token" data key, but isn't labelled as a + // service token (e.g. an S3/Pterodactyl credentials Secret) must never be treated as one. + Secret unlabelled = new SecretBuilder() + .withMetadata(new ObjectMetaBuilder() + .withName("s3-creds") + .withNamespace("bluemap-acme") + .build()) + .withData(Map.of( + FabricPushTokenRepository.TOKEN_DATA_KEY, + Base64.getEncoder().encodeToString("not-a-push-token".getBytes(StandardCharsets.UTF_8)))) + .build(); + client.secrets().inNamespace("bluemap-acme").resource(unlabelled).create(); + + assertTrue(repository().resolveNamespace("not-a-push-token").isEmpty()); + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryPushTokenRepository.java b/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryPushTokenRepository.java new file mode 100644 index 0000000..e2a6696 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryPushTokenRepository.java @@ -0,0 +1,41 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * An in-memory {@link PushTokenRepository} fake for {@code PushControllerTest} -- a plain map is + * enough here (unlike {@link FabricPushTokenRepository}) since proving the constant-time, + * exhaustive-scan comparison itself is that class's own test's job, not the controller's. + */ +final class InMemoryPushTokenRepository implements PushTokenRepository { + + private final Map tokenToNamespace = new HashMap<>(); + + void put(String token, String namespace) { + tokenToNamespace.put(token, namespace); + } + + @Override + public Optional resolveNamespace(String rawToken) { + return Optional.ofNullable(rawToken == null ? null : tokenToNamespace.get(rawToken)); + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryWorldSourceRepository.java b/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryWorldSourceRepository.java new file mode 100644 index 0000000..b769ebb --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/push/InMemoryWorldSourceRepository.java @@ -0,0 +1,63 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import net.onelitefeather.apus.api.rest.worldsource.WorldSourceRepository; +import net.onelitefeather.apus.operator.api.WorldSource; + +/** + * An in-memory, namespace-partitioned {@link WorldSourceRepository} fake for {@code + * PushControllerTest} -- a separate, local copy of the equivalent fake in the {@code + * worldsource} test package rather than reusing it, since that one is package-private there. + */ +final class InMemoryWorldSourceRepository implements WorldSourceRepository { + + private final List items = new ArrayList<>(); + + void put(String namespace, WorldSource source) { + items.add(new Namespaced(namespace, source)); + } + + @Override + public List list(String namespace) { + return items.stream() + .filter(item -> item.namespace().equals(namespace)) + .map(Namespaced::resource) + .toList(); + } + + @Override + public Optional find(String namespace, String name) { + return items.stream() + .filter(item -> item.namespace().equals(namespace) + && item.resource().getMetadata().getName().equals(name)) + .map(Namespaced::resource) + .findFirst(); + } + + @Override + public WorldSource create(String namespace, WorldSource source) { + put(namespace, source); + return source; + } + + private record Namespaced(String namespace, WorldSource resource) {} +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/push/PushControllerTest.java b/api/src/test/java/net/onelitefeather/apus/api/rest/push/PushControllerTest.java new file mode 100644 index 0000000..4204da2 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/push/PushControllerTest.java @@ -0,0 +1,153 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.push; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import net.onelitefeather.apus.api.rest.ingest.InMemoryWorldIngestRepository; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import net.onelitefeather.apus.api.rest.support.NotFoundException; +import net.onelitefeather.apus.operator.api.WorldSource; +import org.junit.jupiter.api.Test; + +/** + * The abuse cases matter more than the good path here (task brief) -- this is the one endpoint in + * the whole module that authenticates via a bare secret in the URL rather than a JWT, so most of + * these tests are deliberately about what happens with a wrong, foreign, or absent token, not + * just the happy path. + */ +class PushControllerTest { + + private final InMemoryPushTokenRepository tokenRepository = new InMemoryPushTokenRepository(); + private final InMemoryWorldSourceRepository sourceRepository = new InMemoryWorldSourceRepository(); + private final InMemoryWorldIngestRepository ingestRepository = new InMemoryWorldIngestRepository(); + private final PushController controller = new PushController(tokenRepository, sourceRepository, ingestRepository); + + private static WorldSource pushSource(String name, String... worldNames) { + WorldSource source = new WorldSource(); + source.getMetadata().setName(name); + source.getSpec().setType("push"); + for (String worldName : worldNames) { + var selector = new WorldSource.WorldSelector(); + selector.setName(worldName); + source.getSpec().getWorlds().add(selector); + } + return source; + } + + @Test + void validTokenAndSourceCreatesOneWorldIngestPerConfiguredWorld() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + sourceRepository.put("bluemap-acme", pushSource("survival", "world", "world_nether")); + + var response = controller.report("acme-secret-token", new PushReportRequest("survival", "backup-42")); + + assertEquals(201, response.getStatus().getCode()); + assertEquals(2, response.body().worldIngests().size()); + assertEquals(2, ingestRepository.forNamespace("bluemap-acme").size()); + var ingest = ingestRepository.forNamespace("bluemap-acme").get(0); + assertEquals("survival", ingest.getSpec().getSourceRef().getName()); + assertEquals("backup-42", ingest.getSpec().getSourceVersion()); + } + + @Test + void unknownTokenIsNotFound() { + sourceRepository.put("bluemap-acme", pushSource("survival", "world")); + + assertThrows( + NotFoundException.class, + () -> controller.report("this-token-does-not-exist", new PushReportRequest("survival", "v1"))); + } + + @Test + void blankTokenIsNotFound() { + assertThrows(NotFoundException.class, () -> controller.report("", new PushReportRequest("survival", "v1"))); + } + + @Test + void tokenForOneTenantCannotReachAnotherTenantsSourceByName() { + // "carol-secret" only authorizes bluemap-acme -- globex-survival exists, but under a + // different namespace this token was never issued for. + tokenRepository.put("carol-secret", "bluemap-acme"); + sourceRepository.put("bluemap-globex", pushSource("globex-survival", "world")); + + assertThrows( + NotFoundException.class, + () -> controller.report("carol-secret", new PushReportRequest("globex-survival", "v1"))); + } + + @Test + void aTokenValidForOneNamespaceNeverCreatesAnIngestInAnotherNamespace() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + tokenRepository.put("globex-secret-token", "bluemap-globex"); + sourceRepository.put("bluemap-acme", pushSource("survival", "world")); + sourceRepository.put("bluemap-globex", pushSource("survival", "world")); + + controller.report("acme-secret-token", new PushReportRequest("survival", "v1")); + + assertEquals(1, ingestRepository.forNamespace("bluemap-acme").size()); + assertTrue(ingestRepository.forNamespace("bluemap-globex").isEmpty()); + } + + @Test + void unknownSourceNameForAValidTokenIsNotFound() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + + assertThrows( + NotFoundException.class, + () -> controller.report("acme-secret-token", new PushReportRequest("no-such-source", "v1"))); + } + + @Test + void aSourceThatIsNotOfTypePushIsNotFoundEvenWithAValidToken() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + WorldSource s3Source = new WorldSource(); + s3Source.getMetadata().setName("survival"); + s3Source.getSpec().setType("s3"); + sourceRepository.put("bluemap-acme", s3Source); + + assertThrows( + NotFoundException.class, + () -> controller.report("acme-secret-token", new PushReportRequest("survival", "v1"))); + } + + @Test + void aSourceWithNoConfiguredWorldsIsRejected() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + sourceRepository.put("bluemap-acme", pushSource("survival")); + + assertThrows( + BadRequestException.class, + () -> controller.report("acme-secret-token", new PushReportRequest("survival", "v1"))); + } + + @Test + void missingSourceNameOrVersionIsRejected() { + tokenRepository.put("acme-secret-token", "bluemap-acme"); + sourceRepository.put("bluemap-acme", pushSource("survival", "world")); + + assertThrows( + BadRequestException.class, () -> controller.report("acme-secret-token", new PushReportRequest(null, "v1"))); + assertThrows( + BadRequestException.class, + () -> controller.report("acme-secret-token", new PushReportRequest("survival", null))); + assertThrows(BadRequestException.class, () -> controller.report("acme-secret-token", null)); + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/upload/InMemoryWorldSourceRepository.java b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/InMemoryWorldSourceRepository.java new file mode 100644 index 0000000..849c8c8 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/InMemoryWorldSourceRepository.java @@ -0,0 +1,59 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import net.onelitefeather.apus.api.rest.worldsource.WorldSourceRepository; +import net.onelitefeather.apus.operator.api.WorldSource; + +/** An in-memory, namespace-partitioned {@link WorldSourceRepository} fake for {@code UploadControllerTest}. */ +final class InMemoryWorldSourceRepository implements WorldSourceRepository { + + private final List items = new ArrayList<>(); + + void put(String namespace, WorldSource source) { + items.add(new Namespaced(namespace, source)); + } + + @Override + public List list(String namespace) { + return items.stream() + .filter(item -> item.namespace().equals(namespace)) + .map(Namespaced::resource) + .toList(); + } + + @Override + public Optional find(String namespace, String name) { + return items.stream() + .filter(item -> item.namespace().equals(namespace) + && item.resource().getMetadata().getName().equals(name)) + .map(Namespaced::resource) + .findFirst(); + } + + @Override + public WorldSource create(String namespace, WorldSource source) { + put(namespace, source); + return source; + } + + private record Namespaced(String namespace, WorldSource resource) {} +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceIntegrationTest.java b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceIntegrationTest.java new file mode 100644 index 0000000..13f7aa0 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceIntegrationTest.java @@ -0,0 +1,239 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.security.SecureRandom; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.MinIOContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3Configuration; +import software.amazon.awssdk.services.s3.model.CreateBucketRequest; +import software.amazon.awssdk.services.s3.model.NoSuchUploadException; +import software.amazon.awssdk.services.s3.presigner.S3Presigner; + +/** + * Drives real HTTP requests against a real MinIO instance to answer, empirically rather than by + * reading AWS SDK documentation, the question the phase 6 task brief poses: which of {@code POST + * /api/uploads}'s stated limits ("eng begrenzt: auf das Staging-Präfix genau dieses Mandanten, + * mit kurzer Gültigkeit und einer Größenbegrenzung") actually hold up against an adversarial + * client, and which do not. See the phase 6 task report for how each result here is interpreted. + * + *

Excluded from {@code :api:test} (matches {@code TenantIsolationIntegrationTest}'s own + * Docker/{@code *IntegrationTest} exclusion in {@code build.gradle.kts}) -- run explicitly via + * {@code ./gradlew :api:integrationTest}. + */ +@Testcontainers +class MultipartUploadServiceIntegrationTest { + + private static final String BUCKET = "staging"; + + // Generated fresh per test run rather than pinned to a fixed literal, so nothing checked + // into source ever looks like a real credential. Lengths follow MinIO's own + // accessKeyMinLen/secretKeyMinLen (3 / 8 characters) with generous headroom. + private static final String ACCESS_KEY = randomAlphanumeric(20); + private static final String SECRET_KEY = randomAlphanumeric(40); + + @Container + private static final MinIOContainer MINIO = + new MinIOContainer(DockerImageName.parse("minio/minio:RELEASE.2024-11-07T00-52-20Z")) + .withUserName(ACCESS_KEY) + .withPassword(SECRET_KEY); + + private static S3Client s3Client; + private static S3Presigner presigner; + private static final HttpClient HTTP = HttpClient.newHttpClient(); + + /** Generates a random alphanumeric string of {@code length} characters. */ + private static String randomAlphanumeric(int length) { + String alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + SecureRandom random = new SecureRandom(); + StringBuilder value = new StringBuilder(length); + for (int i = 0; i < length; i++) { + value.append(alphabet.charAt(random.nextInt(alphabet.length()))); + } + return value.toString(); + } + + @BeforeAll + static void createClientsAndBucket() { + var credentials = StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY)); + s3Client = S3Client.builder() + .endpointOverride(URI.create(MINIO.getS3URL())) + .region(Region.US_EAST_1) + .credentialsProvider(credentials) + .forcePathStyle(true) + .build(); + presigner = S3Presigner.builder() + .endpointOverride(URI.create(MINIO.getS3URL())) + .region(Region.US_EAST_1) + .credentialsProvider(credentials) + .serviceConfiguration(S3Configuration.builder() + .pathStyleAccessEnabled(true) + .checksumValidationEnabled(false) + .build()) + .build(); + s3Client.createBucket(CreateBucketRequest.builder().bucket(BUCKET).build()); + } + + @AfterAll + static void closeClients() { + s3Client.close(); + presigner.close(); + } + + /** 5 MiB -- the S3/Ceph minimum part size (except the last part), kept small so tests stay fast. */ + private static final long PART_SIZE = 5L * 1024 * 1024; + + private MultipartUploadService service(long maxUploadBytes) { + return new MultipartUploadService(s3Client, presigner, BUCKET, "staging/", PART_SIZE, maxUploadBytes, 900); + } + + @Test + void endToEndUploadLandsExactlyAtTheExpectedKeyWithTheRealUploadedBytes() throws Exception { + long totalSize = PART_SIZE + 1024; // forces two parts: one full 5 MiB, one small tail + var created = service(10L * 1024 * 1024 * 1024).createUpload("bluemap-acme", "survival", "world.zip", totalSize); + + assertEquals("staging/bluemap-acme/survival/" + created.version() + "/world.zip", created.key()); + assertEquals(2, created.parts().size()); + + for (var part : created.parts()) { + putPart(part.url(), randomBytes((int) part.sizeBytes())); + } + + var completed = service(10L * 1024 * 1024 * 1024) + .completeUpload("bluemap-acme", "survival", created.version(), "world.zip", created.uploadId()); + + assertEquals(totalSize, completed.totalBytes()); + assertEquals(created.key(), completed.key()); + var stored = s3Client.getObject(b -> b.bucket(BUCKET).key(created.key())); + assertEquals(totalSize, stored.response().contentLength()); + } + + @Test + void aPresignedPartUrlCannotBeRedirectedToADifferentTenantsKey() throws Exception { + var created = service(10L * 1024 * 1024 * 1024) + .createUpload("bluemap-acme", "survival", "world.zip", PART_SIZE); + String legitimateUrl = created.parts().get(0).url(); + + // Swap the tenant namespace segment in the signed URL's path while keeping every other + // character -- including the whole SigV4 query string -- untouched. If prefix confinement + // were only advisory (e.g. enforced solely by application logic that a modified request + // never goes through), this would succeed. It must instead fail the signature check. + String tamperedUrl = legitimateUrl.replace("/bluemap-acme/", "/bluemap-globex/"); + assertTrue(!tamperedUrl.equals(legitimateUrl), "the tamper must actually change the URL"); + + HttpResponse response = putPartExpectingFailure(tamperedUrl, randomBytes((int) PART_SIZE)); + + assertEquals( + 403, + response.statusCode(), + "S3 must reject a presigned URL whose key was altered after signing, got body: " + response.body()); + } + + @Test + void completeUploadAbortsAndRejectsWhenTheActualUploadedTotalExceedsTheConfiguredMaximum() throws Exception { + // maxUploadBytes deliberately smaller than what actually gets uploaded below -- proves + // enforcement happens against the real ListParts total, not the originally declared size. + long generousDeclaredSize = 3 * PART_SIZE; + long tinyMax = PART_SIZE; // one part's worth -- the upload below will exceed this + + var created = service(generousDeclaredSize).createUpload("bluemap-acme", "survival", "world.zip", generousDeclaredSize); + for (var part : created.parts()) { + putPart(part.url(), randomBytes((int) part.sizeBytes())); + } + + MultipartUploadService strict = service(tinyMax); + assertThrows( + BadRequestException.class, + () -> strict.completeUpload("bluemap-acme", "survival", created.version(), "world.zip", created.uploadId())); + + // The upload must actually be gone (aborted), not just rejected at the API layer -- + // otherwise the uploaded-but-oversized parts would sit in the bucket indefinitely. + assertThrows( + NoSuchUploadException.class, + () -> s3Client.listParts( + b -> b.bucket(BUCKET).key(created.key()).uploadId(created.uploadId()))); + } + + /** + * Confirms empirically (against real MinIO, 2026-08-09) that {@code Content-Length} pinning + * on a presigned {@code UploadPart} request genuinely constrains the byte count a client can + * send for that part: sending more bytes than the part was presigned/sized for is rejected + * with HTTP 403 {@code SignatureDoesNotMatch} before those extra bytes are accepted -- + * {@code Content-Length} was a signed header, and the actual request no longer matches what + * was signed. See the phase 6 task report for the full write-up (including the caveat that + * this was verified against MinIO specifically, not independently against Ceph RGW, the + * production backend design spec §9.1 names -- both implement SigV4 the same way, but that is + * an inference, not a second empirical confirmation). + */ + @Test + void aPartExceedingItsPresignedSizeIsRejectedBeforeItIsAccepted() throws Exception { + long declaredPartSize = PART_SIZE; + var created = service(10L * 1024 * 1024 * 1024).createUpload("bluemap-acme", "survival", "world.zip", declaredPartSize); + String partUrl = created.parts().get(0).url(); + + byte[] oversizedBody = randomBytes((int) (declaredPartSize + 1024)); + HttpResponse response = putPartExpectingFailure(partUrl, oversizedBody); + + assertEquals( + 403, + response.statusCode(), + "an oversized part must be rejected by the signature check, got body: " + response.body()); + assertTrue(response.body().contains("SignatureDoesNotMatch")); + } + + private void putPart(String url, byte[] body) throws Exception { + HttpRequest request = HttpRequest.newBuilder(URI.create(url)) + .PUT(HttpRequest.BodyPublishers.ofByteArray(body)) + .build(); + HttpResponse response = HTTP.send(request, HttpResponse.BodyHandlers.ofString()); + if (response.statusCode() / 100 != 2) { + throw new AssertionError("part upload failed: " + response.statusCode() + " " + response.body()); + } + } + + private HttpResponse putPartExpectingFailure(String url, byte[] body) throws Exception { + HttpRequest request = HttpRequest.newBuilder(URI.create(url)) + .PUT(HttpRequest.BodyPublishers.ofByteArray(body)) + .build(); + return HTTP.send(request, HttpResponse.BodyHandlers.ofString()); + } + + private static byte[] randomBytes(int size) { + byte[] data = new byte[size]; + new java.util.Random(42).nextBytes(data); + return data; + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceTest.java b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceTest.java new file mode 100644 index 0000000..98e643b --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/MultipartUploadServiceTest.java @@ -0,0 +1,145 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import org.junit.jupiter.api.Test; + +/** + * Two kinds of proof, deliberately kept apart: + * + *

    + *
  • {@link #stagingKey} confinement -- pure-function tests, no S3/Micronaut involved at all, + * proving the structural guarantee that a staged object's key can never leave {@code + * ///...} no matter what {@code version}/{@code fileName} an + * adversarial caller supplies. + *
  • request validation -- {@link #createUpload}/{@link #completeUpload} reject malformed + * input *before* ever calling the injected {@link software.amazon.awssdk.services.s3.S3Client}/ + * {@link software.amazon.awssdk.services.s3.presigner.S3Presigner}, so these tests + * construct the service with {@code null} for both -- exercising exactly the code paths + * that never dereference them, without needing a mocking framework this project does not + * otherwise depend on. + *
+ * + *

The good-path proof that a presigned URL really is confined to its signed key/size against a + * real S3-compatible backend is {@code MultipartUploadServiceIntegrationTest}'s job instead (real + * MinIO via Testcontainers, Docker required, excluded from this module's default {@code test} + * task exactly like {@code TenantIsolationIntegrationTest}). + */ +class MultipartUploadServiceTest { + + @Test + void stagingKeyIsAlwaysConfinedToThePrefixNamespaceAndSourceSubtree() { + String key = MultipartUploadService.stagingKey("staging/", "bluemap-acme", "survival", "v1", "world.zip"); + + assertEquals("staging/bluemap-acme/survival/v1/world.zip", key); + } + + @Test + void stagingKeyNormalisesAPrefixMissingItsTrailingSlash() { + String key = MultipartUploadService.stagingKey("staging", "bluemap-acme", "survival", "v1", "world.zip"); + + assertEquals("staging/bluemap-acme/survival/v1/world.zip", key); + } + + @Test + void stagingKeyCannotBeEscapedByAnAdversarialVersionOrFileName() { + // S3 keys have no ".."-traversal semantics, so these are just unusual literal key + // segments -- but the property under test is that the namespace segment is untouched + // regardless: whatever a caller supplies for version/fileName, the object still lives + // strictly under this tenant's own namespace/sourceName subtree, never another tenant's. + String key = MultipartUploadService.stagingKey( + "staging/", "bluemap-acme", "survival", "../../bluemap-globex/other-source", "../../evil.zip"); + + assertTrue( + key.startsWith("staging/bluemap-acme/survival/"), + "key must stay under the caller's own namespace/source subtree, was: " + key); + } + + @Test + void namespaceIsTheOnlyInputThatDeterminesTheTenantPrefix() { + String acmeKey = MultipartUploadService.stagingKey("staging/", "bluemap-acme", "survival", "v1", "world.zip"); + String globexKey = MultipartUploadService.stagingKey("staging/", "bluemap-globex", "survival", "v1", "world.zip"); + + assertTrue(acmeKey.startsWith("staging/bluemap-acme/")); + assertTrue(globexKey.startsWith("staging/bluemap-globex/")); + assertTrue(!acmeKey.equals(globexKey)); + } + + private MultipartUploadService serviceWithoutS3() { + return new MultipartUploadService(null, null, "staging-bucket", "staging/", 67_108_864L, 10_737_418_240L, 900L); + } + + @Test + void createUploadRejectsANonPositiveDeclaredSize() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows( + BadRequestException.class, () -> service.createUpload("bluemap-acme", "survival", "world.zip", 0)); + assertThrows( + BadRequestException.class, () -> service.createUpload("bluemap-acme", "survival", "world.zip", -1)); + } + + @Test + void createUploadRejectsADeclaredSizeAboveTheConfiguredMaximum() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows( + BadRequestException.class, + () -> service.createUpload("bluemap-acme", "survival", "world.zip", 10_737_418_240L + 1)); + } + + @Test + void createUploadRejectsAFileNameWithAPathSeparator() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows( + BadRequestException.class, + () -> service.createUpload("bluemap-acme", "survival", "../evil/world.zip", 1024)); + } + + @Test + void createUploadRejectsABlankFileName() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows(BadRequestException.class, () -> service.createUpload("bluemap-acme", "survival", "", 1024)); + assertThrows(BadRequestException.class, () -> service.createUpload("bluemap-acme", "survival", null, 1024)); + } + + @Test + void completeUploadRejectsAMalformedVersion() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows( + BadRequestException.class, + () -> service.completeUpload("bluemap-acme", "survival", "../../bluemap-globex", "world.zip", "up-1")); + } + + @Test + void completeUploadRejectsABlankUploadId() { + MultipartUploadService service = serviceWithoutS3(); + + assertThrows( + BadRequestException.class, + () -> service.completeUpload("bluemap-acme", "survival", "v1", "world.zip", "")); + } +} diff --git a/api/src/test/java/net/onelitefeather/apus/api/rest/upload/UploadControllerTest.java b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/UploadControllerTest.java new file mode 100644 index 0000000..5b56326 --- /dev/null +++ b/api/src/test/java/net/onelitefeather/apus/api/rest/upload/UploadControllerTest.java @@ -0,0 +1,151 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.api.rest.upload; + +import static org.junit.jupiter.api.Assertions.assertThrows; + +import io.micronaut.security.authentication.Authentication; +import java.util.List; +import java.util.Map; +import net.onelitefeather.apus.api.rest.support.BadRequestException; +import net.onelitefeather.apus.api.rest.support.NotFoundException; +import net.onelitefeather.apus.api.security.ForbiddenException; +import net.onelitefeather.apus.api.security.TenantResolver; +import net.onelitefeather.apus.api.support.PrincipalResolver; +import net.onelitefeather.apus.operator.api.WorldSource; +import org.junit.jupiter.api.Test; + +/** + * Covers everything {@code UploadController} decides *before* delegating to {@link + * MultipartUploadService} -- role/tenant scoping and source lookup, exactly the boundary this + * controller is responsible for. {@link MultipartUploadService}'s own request-shape validation is + * {@code MultipartUploadServiceTest}'s job; a presigned URL's real behaviour against S3 is {@code + * MultipartUploadServiceIntegrationTest}'s (Docker/MinIO, not run here). {@link + * #createDelegatesToTheServiceOnceTheSourceCheckPasses()} proves the wiring between this + * controller and that service without needing a real S3 client, by supplying a request the + * service itself rejects for a *different* reason than anything the controller checks -- proof + * that control genuinely passed through. + */ +class UploadControllerTest { + + private final InMemoryWorldSourceRepository sourceRepository = new InMemoryWorldSourceRepository(); + // No real S3Client/S3Presigner: every test here either fails before the service ever touches + // them, or (createDelegatesToTheServiceOnceTheSourceCheckPasses) fails inside the service for + // a reason unrelated to S3 connectivity -- see MultipartUploadServiceTest's Javadoc for why + // that is a safe thing to construct. + private final MultipartUploadService uploadService = + new MultipartUploadService(null, null, "staging-bucket", "staging/", 67_108_864L, 10_737_418_240L, 900L); + private final UploadController controller = + new UploadController(sourceRepository, uploadService, new PrincipalResolver(), new TenantResolver()); + + private static Authentication operator(String tenant) { + return Authentication.build( + "dave", List.of("tenant-operator"), Map.of(PrincipalResolver.TENANT_CLAIM, tenant)); + } + + private static Authentication viewer(String tenant) { + return Authentication.build("carol", List.of("tenant-viewer"), Map.of(PrincipalResolver.TENANT_CLAIM, tenant)); + } + + private static WorldSource uploadSource(String name) { + WorldSource source = new WorldSource(); + source.getMetadata().setName(name); + source.getSpec().setType("upload"); + return source; + } + + @Test + void createRejectsAViewer() { + sourceRepository.put("bluemap-acme", uploadSource("survival")); + + assertThrows( + ForbiddenException.class, + () -> controller.create(viewer("acme"), new CreateUploadRequest("survival", "world.zip", 1024))); + } + + @Test + void createRejectsAnUnknownSourceName() { + assertThrows( + NotFoundException.class, + () -> controller.create(operator("acme"), new CreateUploadRequest("no-such-source", "world.zip", 1024))); + } + + @Test + void createRejectsASourceThatBelongsToAnotherTenant() { + sourceRepository.put("bluemap-globex", uploadSource("globex-survival")); + + assertThrows( + NotFoundException.class, + () -> controller.create( + operator("acme"), new CreateUploadRequest("globex-survival", "world.zip", 1024))); + } + + @Test + void createRejectsASourceThatIsNotOfTypeUpload() { + WorldSource s3Source = new WorldSource(); + s3Source.getMetadata().setName("survival"); + s3Source.getSpec().setType("s3"); + sourceRepository.put("bluemap-acme", s3Source); + + assertThrows( + NotFoundException.class, + () -> controller.create(operator("acme"), new CreateUploadRequest("survival", "world.zip", 1024))); + } + + @Test + void createRejectsAMissingSourceName() { + assertThrows( + BadRequestException.class, + () -> controller.create(operator("acme"), new CreateUploadRequest(null, "world.zip", 1024))); + assertThrows(BadRequestException.class, () -> controller.create(operator("acme"), null)); + } + + @Test + void createDelegatesToTheServiceOnceTheSourceCheckPasses() { + sourceRepository.put("bluemap-acme", uploadSource("survival")); + + // sizeBytes <= 0 is rejected by MultipartUploadService itself, never by the controller -- + // reaching that specific error proves the controller's own checks (role, source lookup) + // all passed and control reached the service. + assertThrows( + BadRequestException.class, + () -> controller.create(operator("acme"), new CreateUploadRequest("survival", "world.zip", 0))); + } + + @Test + void completeRejectsAViewer() { + sourceRepository.put("bluemap-acme", uploadSource("survival")); + + assertThrows( + ForbiddenException.class, + () -> controller.complete( + viewer("acme"), "upload-1", new CompleteUploadRequest("survival", "v1", "world.zip"))); + } + + @Test + void completeRejectsASourceThatBelongsToAnotherTenant() { + sourceRepository.put("bluemap-globex", uploadSource("globex-survival")); + + assertThrows( + NotFoundException.class, + () -> controller.complete( + operator("acme"), + "upload-1", + new CompleteUploadRequest("globex-survival", "v1", "world.zip"))); + } +} diff --git a/docs/superpowers/specs/2026-08-08-apus-design.md b/docs/superpowers/specs/2026-08-08-apus-design.md index e14690e..3d85eb7 100644 --- a/docs/superpowers/specs/2026-08-08-apus-design.md +++ b/docs/superpowers/specs/2026-08-08-apus-design.md @@ -10,6 +10,39 @@ und erlaubt Bedienung ohne YAML. --- +## 0. Stand der Umsetzung + +*(Ergänzt nach Abschluss von Phase 6 — Einstieg für alle, die neu dazukommen.)* + +**Alle sechs Phasen aus §14 sind gebaut**, einschließlich Phase 6 (Push-Quellen: +`paper-worldpush` sowie der UI-Upload-Weg über `POST /api/uploads`). Render-Kern, +Operator/Ingest mit allen vier Connectoren (`s3`, `pterodactyl`, `push`, `upload`), +Hosting, API/UI/Mandanten und die Push-Quellen liegen alle im Hauptzweig. Die +Modul-Tabelle in §4 spiegelt den heutigen Stand wider (inkl. `hosting`, `api`, `ui`, +`paper-worldpush`). + +**Region-Sharding (Phase 4) wurde nach dem Spike bewusst nicht gebaut.** Der Spike +(`docs/superpowers/spikes/2026-08-09-lowres-sharding-spike.md`) wies Kachel-Korruption +bei gleichzeitig laufenden Shards nach; die Entscheidung fiel zugunsten vertikaler +Skalierung über `render-threads` — siehe §14, Phase 4, für die volle Begründung. +`BlueMapMap.spec.shards` existiert und bleibt bis auf Weiteres auf `1` beschränkt. + +**Bewusst offen gelassene Punkte** (Details in §15): + +- **Identity-Broker nicht ausgewählt.** Die API validiert JWTs gegen einen + konfigurierbaren Issuer; welches Produkt (Keycloak, Zitadel, ...) tatsächlich davor + steht, ist nicht entschieden (§15, Punkt 3). +- **OIDC-Anmeldung nie gegen einen echten Broker getestet.** Die Auth-Tests im + `api`-Modul laufen gegen einen Fake-Kubernetes-Client bzw. selbst ausgestellte + Test-JWTs (§13.2); ein Ende-zu-Ende-Lauf gegen einen echten Identity-Broker (Keycloak/ + Zitadel) hat nie stattgefunden. +- **Speicher-/Save-Fenster von `paper-worldpush` ungetestet gegen einen echten + Paper-Server.** Die Kopierlogik ist per Unit-Test abgedeckt, aber `BukkitSaveCoordinator` + (der eigentliche Autosave-Pause-und-Force-Save-Schritt) wurde nie gegen eine laufende + Paper-Instanz oder mit MockBukkit geprüft, anders als in §13.2 ursprünglich vorgesehen. + +--- + ## 1. Ziel und Abgrenzung ### 1.1 Problem @@ -84,7 +117,7 @@ BlueMap-Version zu verifizieren: │ ▼ ┌──────────────────────┐ - │ world-ingest (ETL) │ Extract → Transform → Load + │ ingest (ETL) │ Extract → Transform → Load └──────────┬───────────┘ ▼ World Bundle in S3 ◄────── Vertrag zwischen Ingest und Render @@ -119,16 +152,22 @@ Alle in einem Gradle-Monorepo `Apus`, mehrmodulig. | Modul | Sprache/Stack | Zweck | |---|---|---| -| `telemetry-addon` | Java 21, BlueMap-Addon | Exponiert Render-Fortschritt als JSON und Prometheus-Metriken | -| `world-ingest` | Java 21, Micronaut | ETL: Connector-SPI, Layout-Erkennung, Bundle-Writer. Läuft als Job | -| `runner-image` | Dockerfile + Entrypoint | BlueMap-CLI + beide Addons + Bundle-Sync | -| `operator` | Java 21, Micronaut + Java Operator SDK | Sechs CRDs, erzeugt Jobs/Deployments/Ingresses/Buckets | -| `api` | Java 21, Micronaut | REST + SSE über den CRs, Log-Aggregation, Auth-Durchsetzung | -| `ui` | Nuxt 4, Vue 3, Tailwind 4, Nuxt UI | Zwei Dashboard-Ebenen | -| `paper-worldpush` | Java 21, Paper-Plugin | Async, inkrementeller Welt-Upload vom laufenden Server | - -`telemetry-addon` und `paper-worldpush` hängen an fremden Versionen (BlueMap bzw. Paper) -und bekommen eine eigene Release-Spur mit eigener Versionsmatrix. +| `telemetry-addon` | Java 25, BlueMap-Addon | Exponiert Render-Fortschritt als JSON und Prometheus-Metriken | +| `ingest` | Java 25 | ETL: Connector-SPI (s3, pterodactyl, push, upload), Layout-Erkennung, Bundle-Writer. Läuft als Job | +| `runner` | Dockerfile + Entrypoint | BlueMap-CLI + beide Addons + Bundle-Sync | +| `hosting` | Dockerfile + Entrypoint | Langlebiger Webserver (BlueMap-CLI im `-w`-Modus); liest gerenderte Karten direkt aus S3 über `BlueMapS3Storage`, Konfiguration per gemountetem ConfigMap statt Umgebungsvariablen | +| `operator` | Java 25, Java Operator SDK (fabric8) | Sechs CRDs (`Tenant`, `WorldSource`, `WorldIngest`, `BlueMapMap`, `BlueMapRender`, `BlueMapHosting`), erzeugt Jobs/Deployments/Ingresses/Buckets/Secrets | +| `api` | Java 25, Micronaut | REST + SSE über den CRs, Log-Aggregation, Auth-Durchsetzung | +| `ui` | Nuxt 4, Vue 3, Tailwind 4, Nuxt UI, VueUse | Zwei Dashboard-Ebenen | +| `paper-worldpush` | Java 25, Paper-Plugin | Async, inkrementeller Welt-Upload vom laufenden Server | + +`telemetry-addon` und `paper-worldpush` hängen an fremden Versionen (BlueMap- bzw. +Paper-API) und bekommen eine eigene Release-Spur mit eigener Versionsmatrix — die +Java-*Sprachversion* (Toolchain, einheitlich 25 für das ganze Monorepo, siehe Root- +`build.gradle.kts`) ist davon unabhängig und gilt für jedes Java-Modul gleichermaßen. +`runner` und `hosting` sind reine Dockerfile/Entrypoint-Images ohne eigenen +Gradle-Anwendungscode (kein Java-Sprachversion-Eintrag oben deshalb); `runner` trägt +lediglich Integrationstests, die den Vertrag mit `ingest` prüfen. --- @@ -720,18 +759,21 @@ Credentials erscheinen nie in CR-Status, Events oder Logs. | Baustein | Vorgehen | |---|---| -| `world-ingest` | Fixture-Archive je Layout (Pterodactyl-`tar.gz`, Bukkit-Split, Vanilla, ZIP mit Unterordner, defektes Archiv) gegen den Layout-Detektor. Reine Unit-Tests | +| `ingest` | Fixture-Archive je Layout (Pterodactyl-`tar.gz`, Bukkit-Split, Vanilla, ZIP mit Unterordner, defektes Archiv) gegen den Layout-Detektor. Reine Unit-Tests, plus MinIO-gestützte Integrationstests je Connector (`s3`, `pterodactyl`, `push`, `upload`) und ein Ende-zu-Ende-Test (`PushIngestEndToEndTest`), der einen kompletten Ingest-Lauf für Push/Upload-Quellen gegen echtes MinIO fährt | | `telemetry-addon` | Contract-Test pro BlueMap-Version: Mini-Welt rendern, `/progress` auf plausible Werte prüfen (deckt den Log-Tail-Weg ab, siehe §7.2). **Offen:** Eine CI-Matrix über unterstützte BlueMap-Versionen als Frühwarnsystem existiert nicht — Phase 1 hat im Repository keinerlei CI-Konfiguration angelegt. Bis dahin muss der Contract-Test vor jedem BlueMap-Upgrade manuell laufen | -| `runner-image` | Integrationstest gegen S3-Testcontainer mit kleiner Welt | -| `operator` | JOSDK `LocallyRunOperatorExtension` gegen k3s via Testcontainers | -| `api` | Micronaut-Tests gegen einen Fake-Kubernetes-Client, Auth-Fälle je Rolle | +| `runner` | Integrationstest gegen S3-Testcontainer mit kleiner Welt, inkl. `IngestRenderContractTest` (Ingest → Bundle → Render Ende-zu-Ende) | +| `operator` | JOSDK `LocallyRunOperatorExtension` gegen k3s via Testcontainers, plus `EnableKubernetesMockClient`-Tests je Reconciler | +| `api` | Micronaut-Tests gegen einen Fake-Kubernetes-Client bzw. `EnableKubernetesMockClient`, Auth-Fälle je Rolle. **Offen:** kein Lauf gegen einen echten Identity-Broker (siehe §0/§15, Punkt 3) | | `ui` | Komponententests plus Accessibility-Lint | -| `paper-worldpush` | MockBukkit für die Kopierlogik, zusätzlich ein Lauf gegen einen echten Paper-Server für das Save-Fenster | +| `paper-worldpush` | Unit-Tests für Kopierlogik, Konfiguration und den HTTP-Report-Weg gegen einen lokalen JDK-`HttpServer`-Stub. **Offen:** kein MockBukkit-Test und kein Lauf gegen einen echten Paper-Server für das Save-Fenster (`BukkitSaveCoordinator`) — siehe §0 | | E2E | k3s + S3: kompletter Durchlauf Ingest → Render → Hosting mit Mini-Welt | -**Hinweis zur CRD-Generierung:** Der Fabric8-CRD-Generator ist auf Maven ausgerichtet. Im -Gradle-Monorepo wird er über den Annotation-Processor bzw. eine Gradle-Task eingebunden, -die die `CRDGenerator`-API aufruft. Das ist beim Aufsetzen von Phase 2 zu verifizieren. +**Hinweis zur CRD-Generierung — erledigt.** Der Fabric8-CRD-Generator ist auf Maven +ausgerichtet und bringt keine unterstützte CLI für die genutzte Version (7.8.0). Gelöst +über ein eigenes `crdgen`-Source-Set in `operator/build.gradle.kts` mit einem kleinen +`CrdGeneratorMain`-Einstiegspunkt, der die programmatische `crd-generator-api-v2`/ +`CustomResourceCollector`-API aufruft; eine `generateCrds`-Task erzeugt daraus die sechs +CRD-YAMLs. Siehe §15, Punkt 4. --- @@ -742,7 +784,7 @@ MVP-Bestandteil. ### Phase 1 — Render-Kern *(MVP)* -`telemetry-addon` und `runner-image`. Ergebnis: Ein `docker run` rendert eine Welt aus S3 +`telemetry-addon` und `runner`. Ergebnis: Ein `docker run` rendert eine Welt aus S3 nach S3 und meldet Fortschritt. Vollständig ohne Kubernetes testbar. ### Phase 2 — Operator und Ingest *(MVP)* @@ -800,21 +842,43 @@ ein höherer Wert wird nicht umgesetzt und sollte vom Operator abgelehnt werden. Identity-Broker, `Tenant`-Verwaltung mit Quotas, REST/SSE-API, Vue-Dashboard in zwei Ebenen. -### Phase 6 — Push-Quellen +### Phase 6 — Push-Quellen *(fertig)* -`paper-worldpush` und UI-Upload inklusive Bucket-Notifications. +`paper-worldpush` und UI-Upload. Tatsächlich umgesetzt statt Bucket-Notifications: ein +direkter Completion-Callback vom Schreiber selbst (`POST /api/push/{token}` vom +Paper-Plugin, `POST /api/uploads/{id}/complete` vom UI-Upload-Flow) statt eines +Postfach-artigen Signals aus Ceph oder Polling des Staging-Prefix — siehe §15, Punkt 2, +für die Begründung. --- ## 15. Offene Punkte -1. **Connector-Reihenfolge im MVP.** Angenommen wird: zuerst `s3` und `pterodactyl`, weil beide ohne zusätzliche Client-Software auskommen; `upload` und `push` folgen in Phase 6. Falls das Paper-Plugin der wichtigere Weg ist, verschiebt sich die Reihenfolge — ohne Auswirkung auf die Architektur, da alle Connectoren hinter derselben Schnittstelle liegen. -2. **Bucket-Notifications.** Ob `CephBucketTopic`/`CephBucketNotification` im Cluster nutzbar sind, ist vor Phase 6 zu prüfen. Fallback ist Polling. -3. **Produktwahl Identity-Broker.** Zu Beginn von Phase 5, abgestimmt auf den bestehenden OIDC-Betrieb. -4. **CRD-Generierung unter Gradle.** Vorgehen beim Aufsetzen von Phase 2 verifizieren (§13.2). +1. ~~**Connector-Reihenfolge im MVP.**~~ **Erledigt.** Die angenommene Reihenfolge hat + sich bestätigt: `s3` und `pterodactyl` zuerst (Phase 2), `push` und `upload` in Phase + 6 — das Paper-Plugin hat sich nicht als der wichtigere Weg erwiesen, eine Umsortierung + war nicht nötig. Alle vier Connectoren liegen hinter derselben `WorldSourceConnector`- + Schnittstelle (`ingest/.../connector/`); `IngestConfig`/`IngestMain` verdrahten alle + vier gleichermaßen. +2. ~~**Bucket-Notifications.**~~ **Anders gelöst, nicht mehr offen.** Weder + `CephBucketTopic`/`CephBucketNotification` noch Prefix-Polling wird für Push-Quellen + verwendet: Stattdessen meldet der Schreiber selbst den Abschluss direkt an die API + (`POST /api/push/{token}` vom Paper-Plugin, `POST /api/uploads/{id}/complete` vom + UI-Upload-Flow) — die Prüfung, ob Rook-Notifications im Cluster aktiviert sind, war + damit für den MVP nicht nötig. Bleibt als mögliche spätere Härtung im Hinterkopf, + falls ein Schreiber den Callback verlieren kann (Netzwerkfehler nach dem letzten + Upload, bevor die Meldung rausgeht) und ein zweiter, unabhängiger Erkennungsweg + gewünscht wird. +3. **Produktwahl Identity-Broker.** Weiterhin offen — siehe §0. Die API validiert JWTs + gegen einen konfigurierbaren Issuer, ohne dass ein konkretes Broker-Produkt + (Keycloak/Zitadel) ausgewählt oder gegen einen echten Broker getestet wurde. +4. ~~**CRD-Generierung unter Gradle.**~~ **Erledigt** — siehe §13.2's "Hinweis zur + CRD-Generierung". 5. **`render-mask` und Kanten.** Nur relevant, falls in Phase 4 der Maskenweg statt des eigenen Runners gewählt wird: Ob sich das Auffüllen mit Luft außerhalb der Maske abschalten lässt, ist dann zu prüfen. 6. **Volume-Typ für große Welten.** `emptyDir` genügt bis zu einer Größe, die von der Node-Ausstattung abhängt; darüber ist ein PVC nötig. **Offen:** Diese Grenze wurde in Phase 1 entgegen der ursprünglichen Zusage **nicht** gemessen — es ist eigener Scope, keine bloße Verifikation eines bestehenden Plans. Muss vor Phase 2 nachgeholt werden, bevor der Operator einen Default für die CR festlegt. 7. **Kein belastbares Quota-Signal aus dem Runner-Image.** `BlueMapRenderReconciler` erkennt ein Speicherlimit derzeit heuristisch aus dem Grund/der Meldung des terminierten Render-Pods (Muster wie `QuotaExceeded` oder "quota" kombiniert mit einem S3-Bezug wie `bucket`/`rgw`/`ceph`), gestützt auf `terminationMessagePolicy: FallbackToLogsOnError`, damit überhaupt eine Meldung ankommt. Das bleibt Best-Effort: das Kubelet-Vokabular für den Terminierungsgrund enthält "quota" nie, und die Meldung ist nur ein Log-Ausschnitt ohne Vertrag. Ein belastbares Signal (z. B. ein eigener Exit-Code des Runners für "Quota erschöpft") muss vor einem produktiven Einsatz nachgezogen werden, bevor mehr Verhalten (etwa automatische Benachrichtigungen) darauf aufbaut. +8. **`paper-worldpush`'s Save-Fenster ungetestet gegen einen echten Paper-Server.** §13.2 sah ursprünglich MockBukkit für die Kopierlogik plus einen Lauf gegen einen echten Paper-Server für `BukkitSaveCoordinator`s Autosave-Pause-und-Force-Save-Schritt vor; tatsächlich existiert nur Unit-Testabdeckung für Kopierlogik, Konfiguration und den HTTP-Report-Weg (`HttpPushNotifierTest` gegen einen lokalen `HttpServer`-Stub). Ob das kurze Zeitfenster zwischen `disableAutoSave()`/`forceSave()` und dem Beginn des inkrementellen Kopierens auf einem echten, unter Last laufenden Server tatsächlich einen konsistenten Snapshot liefert, ist vor einem produktiven Einsatz zu verifizieren. +9. **RBAC für den Push-Token-Lookup der API breiter als ideal.** `FabricPushTokenRepository#resolveNamespace` sucht (mangels Tenant-Hinweis im Request) per Label über alle Namespaces nach Service-Token-Secrets; Kubernetes-RBAC kann diesen Zugriff nicht auf das Label einschränken, sodass die schmalste *funktionierende* Berechtigung für das heutige Vorgehen trotzdem `get`/`list` auf **alle** Secrets im Cluster ist (siehe die Klassendoku für die volle Abwägung und einen skizzierten, aber nicht umgesetzten schmaleren Weg über `Tenant`-Enumeration + `get` mit festem Secret-Namen). --- diff --git a/ingest/README.md b/ingest/README.md index 8cd822e..0329bf1 100644 --- a/ingest/README.md +++ b/ingest/README.md @@ -47,7 +47,7 @@ builds Kubernetes Jobs against -- the ingest equivalent of `runner/README.md`'s | Variable | Required | Default | Meaning | |---|---|---|---| -| `APUS_SOURCE_TYPE` | yes | — | `s3` or `pterodactyl`. `upload`/`push` are recognised by `WorldSource.spec.type` but have no connector yet (phase 6) -- an unsupported value fails fast rather than being guessed at | +| `APUS_SOURCE_TYPE` | yes | — | `s3`, `pterodactyl`, `push`, or `upload` -- an unsupported value fails fast rather than being guessed at | | `APUS_WORLD_NAME` | yes | — | The world's folder name at the source, e.g. `world` | | `APUS_LAYOUT` | no | `auto` | `auto`, `vanilla`, or `bukkit`. `auto` lets `LayoutDetector` decide; any other value forces that layout and fails detection rather than falling back if the fetched data doesn't actually match it | | `APUS_SOURCE_VERSION` | yes | — | The exact source version id to fetch, as previously resolved by `WorldSourceReconciler`'s `discover()` poll (task 6) and recorded on the owning `WorldIngest.spec.sourceVersion`. This job never calls `discover()` itself -- see "Design notes" below | @@ -74,6 +74,12 @@ builds Kubernetes Jobs against -- the ingest equivalent of `runner/README.md`'s | `APUS_PTERODACTYL_SERVER_ID` | yes, if pterodactyl | — | Server identifier (short id) | | `APUS_PTERODACTYL_API_KEY` | yes, if pterodactyl | — | Client API key (`ptlc_...`) | | `APUS_PTERODACTYL_WORLD_PATHS` | yes, if pterodactyl | — | Comma-separated top-level archive paths that make up the world, e.g. `world,world_nether,world_the_end` | +| `APUS_SOURCE_STAGING_BUCKET` | yes, if `push` or `upload` | — | Bucket the staged object was written to -- by `paper-worldpush` (its own tenant-scoped credentials) for `push`, or by the browser completing a presigned multipart upload for `upload`. Both types share this one env var contract since only one connector runs per job -- see `AbstractStagedSourceConnector` | +| `APUS_SOURCE_STAGING_ENDPOINT` | no | AWS default | Staging S3-compatible endpoint | +| `APUS_SOURCE_STAGING_PREFIX` | no | `""` | Prefix under which the staged object lives; the object fetched is `prefix + APUS_SOURCE_VERSION` -- there is no `discover()` for these types, the version id is already known (it is the very same id that was used as the staged object's key suffix when it was written; see "Design notes") | +| `APUS_SOURCE_STAGING_ACCESS_KEY` | no | credential chain | Staging access key; if unset, falls back to the AWS SDK default credentials chain | +| `APUS_SOURCE_STAGING_SECRET_KEY` | no | credential chain | Staging secret key | +| `APUS_SOURCE_STAGING_REGION` | no | `us-east-1` | Staging region | Missing a required variable (including the source-specific ones for the chosen `APUS_SOURCE_TYPE`) aborts with a clear `[apus-ingest] ERROR: is required but was not set.` @@ -132,14 +138,21 @@ polling operation -- it belongs to `WorldSourceReconciler` (task 6), which resol new version" on a schedule and records the chosen id on `WorldIngest.spec.sourceVersion`. The job itself only ever calls `fetch()` for the one version it was told to fetch (`APUS_SOURCE_VERSION`); it never lists what's available. This keeps a single ingest run deterministic and keeps the -"what's new" decision in exactly one place. +"what's new" decision in exactly one place. Push-style sources (`push`, `upload`) go one step +further: `discover()` *always* returns an empty list for them (`AbstractStagedSourceConnector`), +because nothing ever polls for their versions in the first place -- the `POST /api/uploads` / +`POST /api/push/{token}` endpoint that creates their `WorldIngest` already knows the version id, +since it is the very same id it used as the staged object's key suffix when it wrote (or arranged +for the client to write) the data. `APUS_SOURCE_VERSION` is that id verbatim. ## Integration tests -`S3SourceConnectorTest` (`ingest/src/test/java/net/onelitefeather/apus/ingest/connector/`) starts -a real MinIO container via Testcontainers and therefore needs Docker. Like `runner` and -`operator` do for their own container-based tests, it is **not** part of `./gradlew build` or -`check` -- it is excluded from the default `test` task and runs only via the explicit task below: +`S3SourceConnectorTest`, `PushSourceConnectorTest`, `UploadSourceConnectorTest` +(`ingest/src/test/java/net/onelitefeather/apus/ingest/connector/`) and `PushIngestEndToEndTest` +(`ingest/src/test/java/net/onelitefeather/apus/ingest/`) all start a real MinIO container via +Testcontainers and therefore need Docker. Like `runner` and `operator` do for their own +container-based tests, none of them are part of `./gradlew build` or `check` -- all four are +excluded from the default `test` task and run only via the explicit task below: ```bash ./gradlew :ingest:integrationTest @@ -150,8 +163,13 @@ Every other test in this module (`IngestConfigTest`, `IngestMainTest`, `Throttle `PterodactylConnectorTest`, `ArchivesTest`, `TarStreamReaderTest`) runs Docker-free as part of the routine `./gradlew :ingest:test`. -A full source-to-bundle-to-render end-to-end test (ingest a Bukkit-layout world fixture against -real MinIO, check the resulting manifest, then start a real render against the produced bundle -with the `runner` image) lives in `runner`'s `:runner:integrationTest` -(`IngestRenderContractTest`), not here -- proving the contract between this module's output and -`runner`'s input needs both modules in the same test. +`PushIngestEndToEndTest` is the proof that `push`/`upload` ingest works end to end, not just that +`PushSourceConnector`/`UploadSourceConnector` behave correctly in isolation: it stages a world +archive in a MinIO prefix (as `paper-worldpush` or a completed presigned upload would leave it), +runs `IngestMain.run` with `APUS_SOURCE_TYPE=push` (and again with `upload`), and asserts the +resulting bundle's manifest and region file are actually there. A separate full +source-to-bundle-to-render end-to-end test (ingest a Bukkit-layout world fixture against real +MinIO, check the resulting manifest, then start a real render against the produced bundle with the +`runner` image) lives in `runner`'s `:runner:integrationTest` (`IngestRenderContractTest`), not +here -- proving the contract between this module's output and `runner`'s input needs both modules +in the same test. diff --git a/ingest/build.gradle.kts b/ingest/build.gradle.kts index c508d19..3b99525 100644 --- a/ingest/build.gradle.kts +++ b/ingest/build.gradle.kts @@ -43,23 +43,32 @@ tasks { } } -// S3SourceConnectorTest starts a real MinIO container via Testcontainers and therefore needs -// Docker. Exactly like runner/build.gradle.kts and operator/build.gradle.kts do for their own -// container-based tests, that must not run as part of the routine `./gradlew build`/`check` -- -// it would make every build slow and fail outright on a machine without Docker. Excluded from -// the default `test` task and exposed only via the explicit `integrationTest` task below. See -// ingest/README.md for how to run it. +// S3SourceConnectorTest, PushSourceConnectorTest, UploadSourceConnectorTest and (phase 6 task 3) +// PushIngestEndToEndTest -- the last one drives the whole IngestMain flow rather than one +// connector method, proving push/upload ingest end to end -- all start a real MinIO container via +// Testcontainers and therefore need Docker. Exactly like runner/build.gradle.kts and +// operator/build.gradle.kts do for their own container-based tests, that must not run as part of +// the routine `./gradlew build`/`check` -- it would make every build slow and fail outright on a +// machine without Docker. Excluded from the default `test` task and exposed only via the explicit +// `integrationTest` task below. See ingest/README.md for how to run it. tasks.test { exclude("**/S3SourceConnectorTest.class") + exclude("**/PushSourceConnectorTest.class") + exclude("**/UploadSourceConnectorTest.class") + exclude("**/PushIngestEndToEndTest.class") } val integrationTest by tasks.registering(Test::class) { group = "verification" - description = "Runs S3SourceConnectorTest against a real MinIO container via Testcontainers. " + + description = "Runs the MinIO-backed connector tests (S3SourceConnectorTest, PushSourceConnectorTest, " + + "UploadSourceConnectorTest, PushIngestEndToEndTest) against a real MinIO container via Testcontainers. " + "Requires Docker. Not part of build/check." testClassesDirs = sourceSets.test.get().output.classesDirs classpath = sourceSets.test.get().runtimeClasspath include("**/S3SourceConnectorTest.class") + include("**/PushSourceConnectorTest.class") + include("**/UploadSourceConnectorTest.class") + include("**/PushIngestEndToEndTest.class") timeout.set(Duration.ofMinutes(5)) outputs.upToDateWhen { false } } diff --git a/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestConfig.java b/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestConfig.java index 262f7b2..c212272 100644 --- a/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestConfig.java +++ b/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestConfig.java @@ -22,7 +22,9 @@ import java.util.Map; import java.util.Set; import net.onelitefeather.apus.ingest.connector.PterodactylConnector; +import net.onelitefeather.apus.ingest.connector.PushSourceConnector; import net.onelitefeather.apus.ingest.connector.S3SourceConnector; +import net.onelitefeather.apus.ingest.connector.UploadSourceConnector; /** * The ingest job's complete configuration, read from environment variables and validated eagerly. @@ -93,9 +95,26 @@ public final class IngestConfig { public static final String ENV_PTERODACTYL_API_KEY = "APUS_PTERODACTYL_API_KEY"; public static final String ENV_PTERODACTYL_WORLD_PATHS = "APUS_PTERODACTYL_WORLD_PATHS"; + // -- Source-specific: push / upload (both are "staged" sources -- see + // AbstractStagedSourceConnector). The data is already sitting in a staging prefix in S3 + // before this job ever starts: paper-worldpush writes it directly with its own tenant-scoped + // credentials (type "push"), or the UI completes a presigned multipart upload to the same + // kind of prefix (type "upload"). Only one of the two types runs per job (APUS_SOURCE_TYPE + // picks exactly one connector), so both share one env var contract rather than duplicating it + // per type -- whichever connector is selected reads the same "staging" values. -- + public static final String ENV_SOURCE_STAGING_ENDPOINT = "APUS_SOURCE_STAGING_ENDPOINT"; + public static final String ENV_SOURCE_STAGING_BUCKET = "APUS_SOURCE_STAGING_BUCKET"; + public static final String ENV_SOURCE_STAGING_PREFIX = "APUS_SOURCE_STAGING_PREFIX"; + public static final String ENV_SOURCE_STAGING_ACCESS_KEY = "APUS_SOURCE_STAGING_ACCESS_KEY"; + public static final String ENV_SOURCE_STAGING_SECRET_KEY = "APUS_SOURCE_STAGING_SECRET_KEY"; + public static final String ENV_SOURCE_STAGING_REGION = "APUS_SOURCE_STAGING_REGION"; + private static final String TYPE_S3 = "s3"; private static final String TYPE_PTERODACTYL = "pterodactyl"; - private static final Set SUPPORTED_SOURCE_TYPES = Set.of(TYPE_S3, TYPE_PTERODACTYL); + private static final String TYPE_PUSH = "push"; + private static final String TYPE_UPLOAD = "upload"; + private static final Set SUPPORTED_SOURCE_TYPES = + Set.of(TYPE_S3, TYPE_PTERODACTYL, TYPE_PUSH, TYPE_UPLOAD); private static final String AUTO_LAYOUT = "auto"; private static final String DEFAULT_S3_REGION = "us-east-1"; @@ -175,8 +194,7 @@ public static IngestConfig fromEnv(Map env) { String sourceType = requireNonBlank(env, ENV_SOURCE_TYPE); if (!SUPPORTED_SOURCE_TYPES.contains(sourceType)) { throw new ConfigurationException("Unsupported " + ENV_SOURCE_TYPE + " '" + sourceType - + "': this image implements only " + SUPPORTED_SOURCE_TYPES - + ". The push sources ('upload', 'push') have no connector yet -- see the phase 2b plan."); + + "': this image implements only " + SUPPORTED_SOURCE_TYPES + "."); } String worldName = requireNonBlank(env, ENV_WORLD_NAME); @@ -206,6 +224,7 @@ public static IngestConfig fromEnv(Map env) { switch (sourceType) { case TYPE_S3 -> s3SourceConfig(env, maxArchiveTotalBytes, maxArchiveEntries); case TYPE_PTERODACTYL -> pterodactylSourceConfig(env, maxArchiveTotalBytes, maxArchiveEntries); + case TYPE_PUSH, TYPE_UPLOAD -> stagingSourceConfig(env, maxArchiveTotalBytes, maxArchiveEntries); default -> throw new IllegalStateException("unreachable: " + sourceType); }; @@ -254,6 +273,27 @@ private static Map pterodactylSourceConfig( return config; } + /** + * Builds the connector config for both push-style sources ({@code push}, {@code upload}). + * Both are handled by {@code AbstractStagedSourceConnector} (via {@link PushSourceConnector} + * or {@link UploadSourceConnector}, chosen by {@code IngestMain} from {@code sourceType}), + * whose config keys are identical between the two subclasses -- only one of the two runs per + * job, so borrowing the constants off {@code PushSourceConnector} here is equivalent to using + * {@code UploadSourceConnector}'s. + */ + private static Map stagingSourceConfig( + Map env, long maxArchiveTotalBytes, long maxArchiveEntries) { + Map config = new LinkedHashMap<>(); + config.put(PushSourceConnector.CONFIG_BUCKET, requireNonBlank(env, ENV_SOURCE_STAGING_BUCKET)); + putIfPresent(config, PushSourceConnector.CONFIG_ENDPOINT, env.get(ENV_SOURCE_STAGING_ENDPOINT)); + putIfPresent(config, PushSourceConnector.CONFIG_PREFIX, env.get(ENV_SOURCE_STAGING_PREFIX)); + putIfPresent(config, PushSourceConnector.CONFIG_ACCESS_KEY_ID, env.get(ENV_SOURCE_STAGING_ACCESS_KEY)); + putIfPresent(config, PushSourceConnector.CONFIG_SECRET_ACCESS_KEY, env.get(ENV_SOURCE_STAGING_SECRET_KEY)); + putIfPresent(config, PushSourceConnector.CONFIG_REGION, env.get(ENV_SOURCE_STAGING_REGION)); + putArchiveLimits(config, maxArchiveTotalBytes, maxArchiveEntries); + return config; + } + private static void putArchiveLimits(Map config, long maxArchiveTotalBytes, long maxArchiveEntries) { config.put(net.onelitefeather.apus.ingest.connector.Archives.CONFIG_MAX_TOTAL_BYTES, Long.toString(maxArchiveTotalBytes)); diff --git a/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestMain.java b/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestMain.java index 18cbce2..f35d303 100644 --- a/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestMain.java +++ b/ingest/src/main/java/net/onelitefeather/apus/ingest/IngestMain.java @@ -23,8 +23,10 @@ import java.time.Instant; import java.util.Map; import net.onelitefeather.apus.ingest.connector.PterodactylConnector; +import net.onelitefeather.apus.ingest.connector.PushSourceConnector; import net.onelitefeather.apus.ingest.connector.S3SourceConnector; import net.onelitefeather.apus.ingest.connector.SourceVersion; +import net.onelitefeather.apus.ingest.connector.UploadSourceConnector; import net.onelitefeather.apus.ingest.connector.WorldSourceConnector; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; @@ -140,6 +142,8 @@ private static WorldSourceConnector selectConnector(String sourceType) { return switch (sourceType) { case "s3" -> new S3SourceConnector(); case "pterodactyl" -> new PterodactylConnector(); + case "push" -> new PushSourceConnector(); + case "upload" -> new UploadSourceConnector(); // IngestConfig.fromEnv already rejects any other value; reaching this would mean the // two disagree about which source types are supported. default -> throw new IllegalStateException("unsupported source type: " + sourceType); diff --git a/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnector.java b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnector.java new file mode 100644 index 0000000..070f417 --- /dev/null +++ b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnector.java @@ -0,0 +1,146 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.ResponseInputStream; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3ClientBuilder; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.model.GetObjectResponse; + +/** + * Shared extract logic for the two push-style sources ({@link PushSourceConnector}, {@link + * UploadSourceConnector}). Both stage their payload as a single object under a prefix in S3 + * before an ingest job ever starts -- a browser upload completing a presigned multipart upload, + * or a Paper server writing directly with its own tenant-scoped credentials (design spec §6.1, + * §11.1). Neither reports versions of its own: whatever created the {@code WorldIngest} (the + * {@code POST /api/uploads}/{@code POST /api/push/{token}} endpoints under the {@code api} + * module) already knows the version id, because it is the very same id it used as the staged + * object's key suffix when it wrote (or arranged for the client to write) the data. + * + *

{@link #discover} therefore always returns an empty list, matching the {@link + * WorldSourceConnector#discover} contract for push sources verbatim. {@link #fetch} is otherwise + * identical to {@link S3SourceConnector#fetch}: the staged object at {@code prefix + + * version.id()} is either extracted (recognised archive extension) or copied as a single raw + * file -- "der Weg ist ähnlich, nur die Herkunft der Version unterscheidet sich" (phase 6 task + * brief). This class intentionally duplicates that small amount of S3-plumbing from {@link + * S3SourceConnector} rather than refactoring it to share code: {@link S3SourceConnector} already + * ships with its own passing test suite, and reaching into it here would risk that class for the + * sake of ~40 lines saved. + */ +abstract class AbstractStagedSourceConnector implements WorldSourceConnector { + + public static final String CONFIG_ENDPOINT = "endpoint"; + public static final String CONFIG_BUCKET = "bucket"; + public static final String CONFIG_PREFIX = "prefix"; + public static final String CONFIG_ACCESS_KEY_ID = "accessKeyId"; + public static final String CONFIG_SECRET_ACCESS_KEY = "secretAccessKey"; + public static final String CONFIG_REGION = "region"; + + private static final String DEFAULT_REGION = "us-east-1"; + + @Override + public final List discover(Map config) { + return Collections.emptyList(); + } + + @Override + public final void fetch(Map config, SourceVersion version, Path workDir) { + try (S3Client client = buildClient(config)) { + fetch(client, config, version, workDir); + } + } + + /** Same as {@link #fetch(Map, SourceVersion, Path)} but against an already-built client, for testing. */ + void fetch(S3Client client, Map config, SourceVersion version, Path workDir) { + String bucket = require(config, CONFIG_BUCKET); + String prefix = normalisePrefix(config.get(CONFIG_PREFIX)); + String key = prefix + version.id(); + + GetObjectRequest request = + GetObjectRequest.builder().bucket(bucket).key(key).build(); + try (ResponseInputStream object = client.getObject(request)) { + if (Archives.isArchive(key)) { + Archives.extract(key, object, workDir, Archives.limitsFrom(config)); + } else { + Path target = workDir.resolve(fileNameOf(key)); + Files.copy(object, target, StandardCopyOption.REPLACE_EXISTING); + } + } catch (IOException e) { + throw new UncheckedIOException("failed to fetch staged object " + key, e); + } + } + + private static S3Client buildClient(Map config) { + S3ClientBuilder builder = S3Client.builder() + .region(Region.of(config.getOrDefault(CONFIG_REGION, DEFAULT_REGION))) + .credentialsProvider(credentialsProvider(config)); + String endpoint = config.get(CONFIG_ENDPOINT); + if (endpoint != null && !endpoint.isBlank()) { + // S3-compatible stores (MinIO, Rook/Ceph, R2, ...) are reached through an endpoint + // override and need path-style bucket addressing rather than AWS's virtual-hosted + // style, which only real S3 DNS resolves -- see S3SourceConnector.buildClient for + // the same reasoning. + builder = builder.endpointOverride(URI.create(endpoint)).forcePathStyle(true); + } + return builder.build(); + } + + private static AwsCredentialsProvider credentialsProvider(Map config) { + String accessKeyId = config.get(CONFIG_ACCESS_KEY_ID); + String secretAccessKey = config.get(CONFIG_SECRET_ACCESS_KEY); + if (accessKeyId != null && secretAccessKey != null) { + return StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKeyId, secretAccessKey)); + } + return DefaultCredentialsProvider.builder().build(); + } + + private static String normalisePrefix(String prefix) { + if (prefix == null || prefix.isBlank()) { + return ""; + } + return prefix.endsWith("/") ? prefix : prefix + "/"; + } + + private static String fileNameOf(String key) { + int lastSlash = key.lastIndexOf('/'); + return lastSlash < 0 ? key : key.substring(lastSlash + 1); + } + + private static String require(Map config, String key) { + String value = config.get(key); + if (value == null || value.isBlank()) { + throw new IllegalArgumentException("missing required staging source config key: " + key); + } + return value; + } +} diff --git a/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/PushSourceConnector.java b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/PushSourceConnector.java new file mode 100644 index 0000000..dfedcc9 --- /dev/null +++ b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/PushSourceConnector.java @@ -0,0 +1,35 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +/** + * A push source: the Paper server plugin ({@code paper-worldpush}) writes its world data + * directly into a staging prefix in S3 using its own tenant-scoped credentials, then calls + * {@code POST /api/push/{token}} to report completion, which creates the {@code WorldIngest} + * this connector's {@link #fetch} eventually runs for. + * + *

All behaviour lives in {@link AbstractStagedSourceConnector}; this class only supplies the + * {@code WorldSourceSpec.type} discriminator. + */ +public final class PushSourceConnector extends AbstractStagedSourceConnector { + + @Override + public String type() { + return "push"; + } +} diff --git a/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnector.java b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnector.java new file mode 100644 index 0000000..fce2e29 --- /dev/null +++ b/ingest/src/main/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnector.java @@ -0,0 +1,34 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +/** + * A browser-driven push source: the UI completes a presigned multipart upload directly against + * S3 ({@code POST /api/uploads}, design spec §11.1), landing the data in the same kind of + * staging prefix {@link PushSourceConnector} consumes for the Paper plugin. + * + *

All behaviour lives in {@link AbstractStagedSourceConnector}; this class only supplies the + * {@code WorldSourceSpec.type} discriminator. + */ +public final class UploadSourceConnector extends AbstractStagedSourceConnector { + + @Override + public String type() { + return "upload"; + } +} diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestConfigTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestConfigTest.java index 9e80612..e147276 100644 --- a/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestConfigTest.java +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestConfigTest.java @@ -94,11 +94,54 @@ void blankValuesAreTreatedAsMissing() { @Test void anUnsupportedSourceTypeIsRejectedRatherThanGuessedAt() { Map env = minimalS3Env(); - env.put(IngestConfig.ENV_SOURCE_TYPE, "upload"); + env.put(IngestConfig.ENV_SOURCE_TYPE, "ftp"); IngestConfig.ConfigurationException e = assertThrows(IngestConfig.ConfigurationException.class, () -> IngestConfig.fromEnv(env)); - assertTrue(e.getMessage().contains("upload")); + assertTrue(e.getMessage().contains("ftp")); + } + + @Test + void pushAndUploadSourceConfigMapEnvVarsToTheSharedStagingConnectorConfigKeys() { + for (String sourceType : new String[] {"push", "upload"}) { + Map env = minimalS3Env(); + env.put(IngestConfig.ENV_SOURCE_TYPE, sourceType); + env.put(IngestConfig.ENV_SOURCE_STAGING_BUCKET, "staging"); + env.put(IngestConfig.ENV_SOURCE_STAGING_ENDPOINT, "http://staging-minio:9000"); + env.put(IngestConfig.ENV_SOURCE_STAGING_PREFIX, "acme/survival/"); + env.put(IngestConfig.ENV_SOURCE_STAGING_ACCESS_KEY, "staging-access"); + env.put(IngestConfig.ENV_SOURCE_STAGING_SECRET_KEY, "staging-secret"); + env.put(IngestConfig.ENV_SOURCE_STAGING_REGION, "eu-central-1"); + + Map sourceConfig = IngestConfig.fromEnv(env).sourceConfig(); + + assertEquals("staging", sourceConfig.get(net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_BUCKET)); + assertEquals( + "http://staging-minio:9000", + sourceConfig.get(net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_ENDPOINT)); + assertEquals( + "acme/survival/", + sourceConfig.get(net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_PREFIX)); + assertEquals( + "staging-access", + sourceConfig.get(net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_ACCESS_KEY_ID)); + assertEquals( + "staging-secret", + sourceConfig.get( + net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_SECRET_ACCESS_KEY)); + assertEquals("eu-central-1", sourceConfig.get(net.onelitefeather.apus.ingest.connector.PushSourceConnector.CONFIG_REGION)); + } + } + + @Test + void missingStagingBucketIsDetectedForPushSources() { + Map env = minimalS3Env(); + env.put(IngestConfig.ENV_SOURCE_TYPE, "push"); + // APUS_SOURCE_STAGING_BUCKET intentionally left unset. + + IngestConfig.ConfigurationException e = + assertThrows(IngestConfig.ConfigurationException.class, () -> IngestConfig.fromEnv(env)); + assertTrue(e.getMessage().contains(IngestConfig.ENV_SOURCE_STAGING_BUCKET)); } @Test diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestMainTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestMainTest.java index 8cbb990..802a6a0 100644 --- a/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestMainTest.java +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/IngestMainTest.java @@ -54,7 +54,7 @@ void missingRequiredVariableExitsNonZeroAndTouchesNothing(@TempDir Path tempDir) void unsupportedSourceTypeExitsNonZeroAndTouchesNothing(@TempDir Path tempDir) { Path workDir = tempDir.resolve("source"); Map env = new LinkedHashMap<>(); - env.put(IngestConfig.ENV_SOURCE_TYPE, "upload"); + env.put(IngestConfig.ENV_SOURCE_TYPE, "ftp"); env.put(IngestConfig.ENV_WORLD_NAME, "world"); env.put(IngestConfig.ENV_SOURCE_VERSION, "v1"); env.put(IngestConfig.ENV_BUNDLE_BUCKET, "bundles"); diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/PushIngestEndToEndTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/PushIngestEndToEndTest.java new file mode 100644 index 0000000..66e4c2d --- /dev/null +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/PushIngestEndToEndTest.java @@ -0,0 +1,209 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.security.SecureRandom; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.zip.ZipEntry; +import java.util.zip.ZipOutputStream; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.MinIOContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.CreateBucketRequest; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; + +/** + * Proves the push/upload ingest path end to end, against a real MinIO instance: world data + * staged in a prefix (as {@code paper-worldpush} or the UI's presigned upload would leave it), + * an ingest job of type {@code push}/{@code upload} run via {@link IngestMain#run}, and a valid + * bundle with a manifest produced from it -- the same "start MinIO via Testcontainers" pattern + * {@code S3SourceConnectorTest} and {@code AbstractStagedSourceConnectorTest} already use in this + * module (see {@code connector/} package), just driving the whole job rather than one connector + * method. + * + *

Before this test existed, {@code IngestConfig} rejected both source types outright (see the + * removed "have no connector yet" message) -- {@code PushSourceConnector} and {@code + * UploadSourceConnector} existed but nothing ever reached them from a real {@code + * APUS_SOURCE_TYPE} value. This is the proof that the wiring (this module's {@code IngestConfig} + * and {@code IngestMain}) now carries a push/upload ingest all the way to a valid bundle, not + * just that the connector class itself behaves correctly in isolation. + * + *

Needs Docker; excluded from {@code :ingest:test} and run only via {@code + * :ingest:integrationTest} -- see {@code ingest/build.gradle.kts} and {@code ingest/README.md}. + */ +@Testcontainers +class PushIngestEndToEndTest { + + private static final String STAGING_BUCKET = "staging"; + private static final String BUNDLE_BUCKET = "bundles"; + + // Generated fresh per test run rather than pinned to a fixed literal, so nothing checked + // into source ever looks like a real credential. Lengths follow MinIO's own + // accessKeyMinLen/secretKeyMinLen (3 / 8 characters) with generous headroom. + private static final String ACCESS_KEY = randomAlphanumeric(20); + private static final String SECRET_KEY = randomAlphanumeric(40); + + @Container + private static final MinIOContainer MINIO = + new MinIOContainer(DockerImageName.parse("minio/minio:RELEASE.2024-11-07T00-52-20Z")) + .withUserName(ACCESS_KEY) + .withPassword(SECRET_KEY); + + private static S3Client sharedClient; + + /** Generates a random alphanumeric string of {@code length} characters. */ + private static String randomAlphanumeric(int length) { + String alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + SecureRandom random = new SecureRandom(); + StringBuilder value = new StringBuilder(length); + for (int i = 0; i < length; i++) { + value.append(alphabet.charAt(random.nextInt(alphabet.length()))); + } + return value.toString(); + } + + @BeforeAll + static void createClientAndBuckets() { + sharedClient = S3Client.builder() + .endpointOverride(URI.create(MINIO.getS3URL())) + .region(Region.US_EAST_1) + .credentialsProvider( + StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY))) + .forcePathStyle(true) + .build(); + sharedClient.createBucket(CreateBucketRequest.builder().bucket(STAGING_BUCKET).build()); + sharedClient.createBucket(CreateBucketRequest.builder().bucket(BUNDLE_BUCKET).build()); + } + + @AfterAll + static void closeClient() { + sharedClient.close(); + } + + /** + * Runs the same scenario for both push-style source types: a Paper server ({@code push}) and + * a browser upload ({@code upload}) stage their world data identically (design spec §6.1, + * §11.1) and are handled by the same {@code AbstractStagedSourceConnector} logic, so both + * must come out the other end of {@code IngestMain} the same way. + */ + @ParameterizedTest(name = "sourceType={0}") + @ValueSource(strings = {"push", "upload"}) + void stagedWorldDataProducesAValidBundleWithManifest(String sourceType, @TempDir Path workDir) throws IOException { + String tenant = "acme"; + String sourceName = "survival-source-" + sourceType; + String worldId = "survival"; + String bundleVersion = "v1"; + String stagingPrefix = tenant + "/" + sourceName + "/"; + String sourceVersionId = "2026-08-09T00-00-00Z.zip"; + + byte[] zip = buildZip(Map.of( + "world/level.dat", "level-dat-bytes", + "world/region/r.0.0.mca", "region-data-bytes")); + sharedClient.putObject( + PutObjectRequest.builder() + .bucket(STAGING_BUCKET) + .key(stagingPrefix + sourceVersionId) + .build(), + RequestBody.fromBytes(zip)); + + Map env = new LinkedHashMap<>(); + env.put(IngestConfig.ENV_SOURCE_TYPE, sourceType); + env.put(IngestConfig.ENV_WORLD_NAME, "world"); + env.put(IngestConfig.ENV_SOURCE_VERSION, sourceVersionId); + env.put(IngestConfig.ENV_BUNDLE_BUCKET, BUNDLE_BUCKET); + env.put(IngestConfig.ENV_BUNDLE_TENANT, tenant); + env.put(IngestConfig.ENV_BUNDLE_SOURCE_NAME, sourceName); + env.put(IngestConfig.ENV_BUNDLE_WORLD_ID, worldId); + env.put(IngestConfig.ENV_BUNDLE_VERSION, bundleVersion); + env.put(IngestConfig.ENV_S3_ENDPOINT, MINIO.getS3URL()); + env.put(IngestConfig.ENV_S3_ACCESS_KEY, ACCESS_KEY); + env.put(IngestConfig.ENV_S3_SECRET_KEY, SECRET_KEY); + env.put(IngestConfig.ENV_MC_VERSION, "1.21.10"); + env.put(IngestConfig.ENV_SOURCE_STAGING_ENDPOINT, MINIO.getS3URL()); + env.put(IngestConfig.ENV_SOURCE_STAGING_BUCKET, STAGING_BUCKET); + env.put(IngestConfig.ENV_SOURCE_STAGING_PREFIX, stagingPrefix); + env.put(IngestConfig.ENV_SOURCE_STAGING_ACCESS_KEY, ACCESS_KEY); + env.put(IngestConfig.ENV_SOURCE_STAGING_SECRET_KEY, SECRET_KEY); + + int exitCode = IngestMain.run(env, workDir.resolve("work")); + + assertEquals(0, exitCode, "ingest of a staged " + sourceType + " source must succeed end to end"); + + String bundlePath = BundlePath.of(tenant, sourceName, worldId, bundleVersion); + String manifestJson = getObjectAsString(BUNDLE_BUCKET, bundlePath + "/manifest.json"); + BundleManifest manifest = BundleManifest.fromJson(manifestJson); + + assertEquals(tenant, manifest.tenant()); + assertEquals(worldId, manifest.worldId()); + assertEquals(bundleVersion, manifest.version()); + assertEquals(sourceType, manifest.source().type()); + assertEquals(sourceVersionId, manifest.source().ref()); + assertEquals("vanilla", manifest.source().detectedLayout()); + assertEquals("1.21.10", manifest.minecraftVersion()); + assertEquals(1, manifest.dimensions().size()); + assertTrue(manifest.sizeBytes() > 0); + assertNotNull(manifest.checksums().manifest()); + + // The region file the manifest describes must actually be present under the bundle path + // -- the manifest is only the commit point, not proof on its own that the data exists. + String regionObject = getObjectAsString( + BUNDLE_BUCKET, manifest.dimensions().get(0).path() + "/region/r.0.0.mca"); + assertEquals("region-data-bytes", regionObject); + } + + private static String getObjectAsString(String bucket, String key) throws IOException { + try (var object = sharedClient.getObject( + GetObjectRequest.builder().bucket(bucket).key(key).build())) { + return new String(object.readAllBytes(), StandardCharsets.UTF_8); + } + } + + private static byte[] buildZip(Map entries) throws IOException { + var buffer = new ByteArrayOutputStream(); + try (var zip = new ZipOutputStream(buffer)) { + for (var entry : entries.entrySet()) { + zip.putNextEntry(new ZipEntry(entry.getKey())); + zip.write(entry.getValue().getBytes(StandardCharsets.UTF_8)); + zip.closeEntry(); + } + } + return buffer.toByteArray(); + } +} diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnectorTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnectorTest.java new file mode 100644 index 0000000..564580a --- /dev/null +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/AbstractStagedSourceConnectorTest.java @@ -0,0 +1,205 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.SecureRandom; +import java.time.Instant; +import java.util.Comparator; +import java.util.HashMap; +import java.util.Map; +import java.util.zip.ZipEntry; +import java.util.zip.ZipOutputStream; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.testcontainers.containers.MinIOContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.CreateBucketRequest; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; + +/** + * Behaviour shared by {@link PushSourceConnectorTest} and {@link UploadSourceConnectorTest}: both + * connectors delegate everything to {@link AbstractStagedSourceConnector}, so both need to prove + * the same three things against a real MinIO instance -- {@code discover} is always empty (push + * semantics), a staged archive is extracted, and a staged plain file is copied as-is. Only the + * {@code type()} discriminator differs between the two connectors, which each subclass's own + * (much smaller) test class asserts on top of what is proven here. + * + *

Mirrors {@code S3SourceConnectorTest}'s own Testcontainers setup deliberately -- see that + * class's Javadoc for why a real MinIO instance is used instead of a hand-rolled stub. + */ +@Testcontainers +abstract class AbstractStagedSourceConnectorTest { + + private static final String BUCKET = "worlds"; + + // Generated fresh per test run rather than pinned to a fixed literal, so nothing checked + // into source ever looks like a real credential. Lengths follow MinIO's own + // accessKeyMinLen/secretKeyMinLen (3 / 8 characters) with generous headroom. + private static final String ACCESS_KEY = randomAlphanumeric(20); + private static final String SECRET_KEY = randomAlphanumeric(40); + + @Container + private static final MinIOContainer MINIO = + new MinIOContainer(DockerImageName.parse("minio/minio:RELEASE.2024-11-07T00-52-20Z")) + .withUserName(ACCESS_KEY) + .withPassword(SECRET_KEY); + + private static S3Client sharedClient; + + /** Generates a random alphanumeric string of {@code length} characters. */ + private static String randomAlphanumeric(int length) { + String alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + SecureRandom random = new SecureRandom(); + StringBuilder value = new StringBuilder(length); + for (int i = 0; i < length; i++) { + value.append(alphabet.charAt(random.nextInt(alphabet.length()))); + } + return value.toString(); + } + + @BeforeAll + static void createClientAndBucket() { + sharedClient = S3Client.builder() + .endpointOverride(URI.create(MINIO.getS3URL())) + .region(Region.US_EAST_1) + .credentialsProvider( + StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY))) + .forcePathStyle(true) + .build(); + sharedClient.createBucket(CreateBucketRequest.builder().bucket(BUCKET).build()); + } + + @AfterAll + static void closeClient() { + sharedClient.close(); + } + + /** The connector under test -- supplied by the concrete subclass. */ + abstract AbstractStagedSourceConnector connector(); + + @Test + void discoverAlwaysReportsNoVersionsRegardlessOfWhatIsStaged() { + String prefix = "discover-test/" + getClass().getSimpleName() + "/"; + putObject(prefix + "some-staged-object.zip", "irrelevant"); + + assertEquals(java.util.List.of(), connector().discover(config(prefix))); + } + + @Test + void fetchOfAZipVersionExtractsItsEntriesIntoTheWorkDirectory(@TempDir Path workDir) throws IOException { + String prefix = "zip-test/" + getClass().getSimpleName() + "/"; + byte[] zip = buildZip(Map.of( + "level.dat", "level-data", + "region/r.0.0.mca", "region-data")); + putObject(prefix + "v1.zip", zip); + + connector() + .fetch( + sharedClient, + config(prefix), + new SourceVersion("v1.zip", "v1.zip", Instant.now(), zip.length), + workDir); + + assertEquals("level-data", Files.readString(workDir.resolve("level.dat"))); + assertEquals("region-data", Files.readString(workDir.resolve("region/r.0.0.mca"))); + } + + @Test + void fetchOfATarGzVersionExtractsItsEntriesIntoTheWorkDirectory(@TempDir Path workDir) throws IOException { + String prefix = "targz-test/" + getClass().getSimpleName() + "/"; + byte[] tarGz = new TestTarBuilder() + .addFile("level.dat", "level-data") + .addFile("region/r.0.0.mca", "region-data") + .toGzippedTarBytes(); + putObject(prefix + "v2.tar.gz", tarGz); + + connector() + .fetch( + sharedClient, + config(prefix), + new SourceVersion("v2.tar.gz", "v2.tar.gz", Instant.now(), tarGz.length), + workDir); + + assertEquals("level-data", Files.readString(workDir.resolve("level.dat"))); + assertEquals("region-data", Files.readString(workDir.resolve("region/r.0.0.mca"))); + } + + @Test + void fetchOfAPlainVersionWritesItAsASingleRawFile(@TempDir Path workDir) throws IOException { + String prefix = "raw-test/" + getClass().getSimpleName() + "/"; + putObject(prefix + "raw-dump.bin", "not-an-archive"); + + connector() + .fetch( + sharedClient, + config(prefix), + new SourceVersion("raw-dump.bin", "raw-dump.bin", Instant.now(), 14), + workDir); + + assertTrue(Files.exists(workDir.resolve("raw-dump.bin"))); + assertEquals("not-an-archive", Files.readString(workDir.resolve("raw-dump.bin"))); + } + + private Map config(String prefix) { + Map config = new HashMap<>(); + config.put(AbstractStagedSourceConnector.CONFIG_BUCKET, BUCKET); + config.put(AbstractStagedSourceConnector.CONFIG_PREFIX, prefix); + return config; + } + + private void putObject(String key, String content) { + putObject(key, content.getBytes(StandardCharsets.UTF_8)); + } + + private void putObject(String key, byte[] content) { + sharedClient.putObject( + PutObjectRequest.builder().bucket(BUCKET).key(key).build(), RequestBody.fromBytes(content)); + } + + private static byte[] buildZip(Map entries) throws IOException { + var buffer = new ByteArrayOutputStream(); + try (var zip = new ZipOutputStream(buffer)) { + for (var entry : entries.entrySet().stream() + .sorted(Map.Entry.comparingByKey(Comparator.naturalOrder())) + .toList()) { + zip.putNextEntry(new ZipEntry(entry.getKey())); + zip.write(entry.getValue().getBytes(StandardCharsets.UTF_8)); + zip.closeEntry(); + } + } + return buffer.toByteArray(); + } +} diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/PushSourceConnectorTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/PushSourceConnectorTest.java new file mode 100644 index 0000000..0493fcd --- /dev/null +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/PushSourceConnectorTest.java @@ -0,0 +1,37 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import org.junit.jupiter.api.Test; + +class PushSourceConnectorTest extends AbstractStagedSourceConnectorTest { + + private final PushSourceConnector connector = new PushSourceConnector(); + + @Override + AbstractStagedSourceConnector connector() { + return connector; + } + + @Test + void typeIsPush() { + assertEquals("push", connector.type()); + } +} diff --git a/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnectorTest.java b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnectorTest.java new file mode 100644 index 0000000..9b1b979 --- /dev/null +++ b/ingest/src/test/java/net/onelitefeather/apus/ingest/connector/UploadSourceConnectorTest.java @@ -0,0 +1,37 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.ingest.connector; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import org.junit.jupiter.api.Test; + +class UploadSourceConnectorTest extends AbstractStagedSourceConnectorTest { + + private final UploadSourceConnector connector = new UploadSourceConnector(); + + @Override + AbstractStagedSourceConnector connector() { + return connector; + } + + @Test + void typeIsUpload() { + assertEquals("upload", connector.type()); + } +} diff --git a/operator/src/main/java/net/onelitefeather/apus/operator/api/TenantStatus.java b/operator/src/main/java/net/onelitefeather/apus/operator/api/TenantStatus.java index 4ebc7a9..9949ec8 100644 --- a/operator/src/main/java/net/onelitefeather/apus/operator/api/TenantStatus.java +++ b/operator/src/main/java/net/onelitefeather/apus/operator/api/TenantStatus.java @@ -27,6 +27,7 @@ public class TenantStatus { private String namespace; private String objectStoreUser; private Long storageUsedBytes; + private String pushTokenSecret; private List conditions = new ArrayList<>(); public String getNamespace() { @@ -53,6 +54,22 @@ public void setStorageUsedBytes(Long storageUsedBytes) { this.storageUsedBytes = storageUsedBytes; } + /** + * The name of the {@code Secret} carrying this tenant's {@code world:push} service token, or + * {@code null} if none has been provisioned yet. Deliberately only the Secret's name (a + * fixed, non-secret constant, {@code PushTokenSecrets.SECRET_NAME}) -- never the token value + * itself, which must never appear in a Custom Resource's status, in an event, or in a log + * line. This field says at most "a token exists, here is where"; reading its value always + * requires a separate, RBAC-guarded {@code Secret} read. + */ + public String getPushTokenSecret() { + return pushTokenSecret; + } + + public void setPushTokenSecret(String pushTokenSecret) { + this.pushTokenSecret = pushTokenSecret; + } + public List getConditions() { return conditions; } diff --git a/operator/src/main/java/net/onelitefeather/apus/operator/tenant/PushTokenSecrets.java b/operator/src/main/java/net/onelitefeather/apus/operator/tenant/PushTokenSecrets.java new file mode 100644 index 0000000..c241c6c --- /dev/null +++ b/operator/src/main/java/net/onelitefeather/apus/operator/tenant/PushTokenSecrets.java @@ -0,0 +1,87 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.operator.tenant; + +import java.security.SecureRandom; +import java.util.Base64; + +/** + * The Kubernetes {@code Secret} shape a tenant's {@code world:push} service token is stored in + * -- shared between the two independent places that need to agree on it: {@link + * TenantReconciler} (which creates the Secret) and {@code + * net.onelitefeather.apus.api.rest.push.FabricPushTokenRepository} in the {@code api} module + * (which reads it to authenticate {@code POST /api/push/{token}}). {@code api} already depends + * on {@code operator} for its CRD types (see {@code api/build.gradle.kts}), so these constants + * live here as the one canonical definition rather than being duplicated (and risking drift, the + * exact failure this phase's task brief calls out between {@code paper-worldpush} and {@code + * api}) on both sides. + * + *

Design decision: tenant-scoped, not {@code WorldSource}-scoped. The design spec + * (§10.3) already settles this: "Service-Tokens sind mandantengebunden" (service tokens are + * tenant-bound) -- deliberately not tied to any individual user login or, by extension, any one + * {@code WorldSource}, so that renaming/recreating a push-type source (or a tenant running + * several of them) never invalidates the one token {@code paper-worldpush} was configured with. + * One Secret per tenant namespace is therefore enough; every {@code push}-type {@code + * WorldSource} in that tenant shares it, exactly like {@link + * net.onelitefeather.apus.api.rest.push.FabricPushTokenRepository#resolveNamespace} already + * assumes (it resolves a token to a *namespace*, not to one specific source). + * + *

Never logged, never in status. {@link #generate()} returns the raw token exactly + * once, to the caller that is about to write it into {@code Secret.stringData} and nowhere else + * -- {@link TenantReconciler} does not log it, and {@code TenantStatus} only ever records that + * the Secret exists (by name; the name is a fixed, non-secret constant), never its value. + */ +public final class PushTokenSecrets { + + /** Label key marking a Secret as a {@code world:push} service token; the only way it is found. */ + public static final String LABEL_KEY = "apus.onelitefeather.net/service-token"; + + /** Label value for {@link #LABEL_KEY} -- see {@link #LABEL_KEY}. */ + public static final String LABEL_VALUE = "world-push"; + + /** The key under {@code Secret.data}/{@code Secret.stringData} holding the raw token. */ + public static final String TOKEN_KEY = "token"; + + /** + * Fixed name every tenant's push-token Secret is created/looked up under, within its own + * namespace ({@code bluemap-}). Fixed (not derived per-{@code WorldSource}) because + * exactly one token exists per tenant -- see the class Javadoc -- and because a fixed name + * is what lets the narrowest working RBAC grant restrict {@code get} to {@code + * resourceNames: ["apus-push-token"]} instead of every Secret in the namespace; see {@code + * FabricPushTokenRepository}'s Javadoc for the full RBAC discussion. + */ + public static final String SECRET_NAME = "apus-push-token"; + + /** 256 bits -- generous for a shared secret that is never brute-forced online (rate-limited by the API). */ + private static final int TOKEN_BYTES = 32; + + private static final SecureRandom RANDOM = new SecureRandom(); + + private PushTokenSecrets() {} + + /** + * Generates a new cryptographically random token, URL-safe and unpadded so it can be used + * verbatim as a URL path segment ({@code POST /api/push/{token}}, exactly how {@code + * HttpPushNotifier} in {@code paper-worldpush} sends it) without any escaping. + */ + public static String generate() { + byte[] bytes = new byte[TOKEN_BYTES]; + RANDOM.nextBytes(bytes); + return Base64.getUrlEncoder().withoutPadding().encodeToString(bytes); + } +} diff --git a/operator/src/main/java/net/onelitefeather/apus/operator/tenant/TenantReconciler.java b/operator/src/main/java/net/onelitefeather/apus/operator/tenant/TenantReconciler.java index b096eb9..fb13439 100644 --- a/operator/src/main/java/net/onelitefeather/apus/operator/tenant/TenantReconciler.java +++ b/operator/src/main/java/net/onelitefeather/apus/operator/tenant/TenantReconciler.java @@ -24,12 +24,15 @@ import io.fabric8.kubernetes.api.model.OwnerReferenceBuilder; import io.fabric8.kubernetes.api.model.Quantity; import io.fabric8.kubernetes.api.model.ResourceQuotaBuilder; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; import io.fabric8.kubernetes.client.KubernetesClient; import io.fabric8.kubernetes.client.dsl.NonDeletingOperation; import io.javaoperatorsdk.operator.api.reconciler.Context; import io.javaoperatorsdk.operator.api.reconciler.ControllerConfiguration; import io.javaoperatorsdk.operator.api.reconciler.Reconciler; import io.javaoperatorsdk.operator.api.reconciler.UpdateControl; +import java.util.HashMap; import java.util.Map; import java.util.Objects; import net.onelitefeather.apus.operator.OperatorConfig; @@ -61,6 +64,16 @@ * {@code ResourceConflict} condition instead of silently adopting -- and thereby leaking the * contents of -- someone else's namespace or storage user. * + *

Push-token Secret: a tenant also gets exactly one {@code world:push} service-token + * Secret ({@link PushTokenSecrets#SECRET_NAME}), created once and never regenerated on later + * reconciles -- unlike every other resource here, it is not safe to rebuild fresh from the + * tenant spec each time, because a client ({@code paper-worldpush}) is configured with its value + * once and keeps using it. See {@link PushTokenSecrets} for why this lives at the tenant level + * rather than per-{@code WorldSource}, and {@code FabricPushTokenRepository} in the {@code api} + * module for how it is read back. The raw token is never logged and never written to {@code + * Tenant.status} -- only {@link net.onelitefeather.apus.operator.api.TenantStatus#getPushTokenSecret()}, + * the Secret's (non-secret, fixed) name, is. + * *

Rook not (yet) installed: {@link #reconcile} checks {@link * io.fabric8.kubernetes.client.Client#supports(Class)} for {@link CephObjectStoreUser} before * touching it. If Rook's {@code CephObjectStoreUser} CRD is not registered on the cluster, the @@ -120,6 +133,15 @@ public UpdateControl reconcile(Tenant tenant, Context context) { return conflict(tenant, "Namespace", namespace); } + Secret existingPushToken = client.secrets() + .inNamespace(namespace) + .withName(PushTokenSecrets.SECRET_NAME) + .get(); + if (existingPushToken != null + && !ownedBySameTenant(existingPushToken.getMetadata().getLabels(), tenantName, tenantUid)) { + return conflict(tenant, "Secret", PushTokenSecrets.SECRET_NAME); + } + // Rook may not be installed yet (e.g. a fresh cluster, or a plain k3s test cluster with // no storage operator at all). supports() asks the API server's discovery document // whether the CRD is registered, rather than probing with a get()/create() call and @@ -151,6 +173,29 @@ public UpdateControl reconcile(Tenant tenant, Context context) { .build()) .createOr(NonDeletingOperation::update); + // The push-token Secret is created exactly once and never touched again on subsequent + // reconciles (no createOr(update) here, deliberately -- see the class Javadoc): every + // other resource above is rebuilt fresh from the tenant spec each time, which is fine + // because none of it is a secret a client already holds. A push token is different -- + // paper-worldpush is configured with the value once and keeps using it; regenerating it + // on every resync (as createOr(update) would, since a freshly-built object here would + // carry brand-new random stringData) would silently break every server already pushing. + if (existingPushToken == null) { + client.secrets() + .inNamespace(namespace) + .resource(new SecretBuilder() + .withNewMetadata() + .withName(PushTokenSecrets.SECRET_NAME) + .withNamespace(namespace) + .withLabels(pushTokenLabels(tenantName, tenantUid)) + .withOwnerReferences(ownerReference) + .endMetadata() + .withStringData(Map.of(PushTokenSecrets.TOKEN_KEY, PushTokenSecrets.generate())) + .build()) + .create(); + } + tenant.getStatus().setPushTokenSecret(PushTokenSecrets.SECRET_NAME); + client.resourceQuotas() .inNamespace(namespace) .resource(new ResourceQuotaBuilder() @@ -261,6 +306,19 @@ private static Map tenantLabels(String tenantName, String tenant return labels; } + /** + * The push-token Secret's labels: the standard tenant-ownership labels every resource here + * carries (so the same {@link #ownedBySameTenant} check applies to it), plus {@link + * PushTokenSecrets#LABEL_KEY}/{@link PushTokenSecrets#LABEL_VALUE} -- the label {@code + * FabricPushTokenRepository} in the {@code api} module actually queries by, since a raw push + * token carries no namespace of its own to look the Secret up by name directly. + */ + private static Map pushTokenLabels(String tenantName, String tenantUid) { + Map labels = new HashMap<>(tenantLabels(tenantName, tenantUid)); + labels.put(PushTokenSecrets.LABEL_KEY, PushTokenSecrets.LABEL_VALUE); + return labels; + } + /** Tenant is cluster-scoped, so a namespace (also cluster-scoped) can safely be owned by it. */ private static OwnerReference tenantOwnerReference(Tenant tenant) { return new OwnerReferenceBuilder() diff --git a/operator/src/test/java/net/onelitefeather/apus/operator/tenant/TenantReconcilerTest.java b/operator/src/test/java/net/onelitefeather/apus/operator/tenant/TenantReconcilerTest.java index 95c496e..c8730cb 100644 --- a/operator/src/test/java/net/onelitefeather/apus/operator/tenant/TenantReconcilerTest.java +++ b/operator/src/test/java/net/onelitefeather/apus/operator/tenant/TenantReconcilerTest.java @@ -25,6 +25,8 @@ import io.fabric8.kubernetes.api.model.NamespaceBuilder; import io.fabric8.kubernetes.api.model.ObjectMetaBuilder; import io.fabric8.kubernetes.api.model.ResourceQuota; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; import io.fabric8.kubernetes.client.KubernetesClient; import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer; @@ -58,6 +60,20 @@ private Tenant tenant(String name, String quota) { return tenant; } + /** + * The fabric8 CRUD mock server does not simulate the real API server's {@code stringData} -> + * base64 {@code data} merge on write, so a Secret created via {@code withStringData(...)} + * (as {@code TenantReconciler} does) is read back with the value still under {@code + * stringData}, not {@code data}, here -- unlike a real cluster. Reading either map keeps + * these tests meaningful under both. + */ + private static String tokenValue(Secret secret) { + if (secret.getStringData() != null && secret.getStringData().get(PushTokenSecrets.TOKEN_KEY) != null) { + return secret.getStringData().get(PushTokenSecrets.TOKEN_KEY); + } + return secret.getData() == null ? null : secret.getData().get(PushTokenSecrets.TOKEN_KEY); + } + private String readyReason(Tenant tenant) { return tenant.getStatus().getConditions().stream() .filter(condition -> Conditions.READY.equals(condition.getType())) @@ -322,4 +338,94 @@ void setsAnOwnerReferenceOnTheNamespacePointingAtTheTenant() { .anyMatch(ref -> "Tenant".equals(ref.getKind())), "namespace must be owned by its Tenant so it is garbage-collected on deletion"); } + + @Test + void createsAPushTokenSecretForANewTenant() { + TenantReconciler reconciler = new TenantReconciler(client, OperatorConfig.defaults()); + + reconciler.reconcile(tenant("friends", "500Gi"), null); + + Secret secret = client.secrets() + .inNamespace("bluemap-friends") + .withName(PushTokenSecrets.SECRET_NAME) + .get(); + assertNotNull(secret, "push-token secret must be created"); + assertEquals( + PushTokenSecrets.LABEL_VALUE, + secret.getMetadata().getLabels().get(PushTokenSecrets.LABEL_KEY), + "must carry the label FabricPushTokenRepository queries by"); + assertNotNull(tokenValue(secret), "token data must be present"); + } + + @Test + void reportsThePushTokenSecretNameInStatus() { + TenantReconciler reconciler = new TenantReconciler(client, OperatorConfig.defaults()); + Tenant tenant = tenant("friends", "500Gi"); + + reconciler.reconcile(tenant, null); + + assertEquals(PushTokenSecrets.SECRET_NAME, tenant.getStatus().getPushTokenSecret()); + } + + @Test + void neverRegeneratesAnExistingPushTokenOnLaterReconciles() { + TenantReconciler reconciler = new TenantReconciler(client, OperatorConfig.defaults()); + Tenant tenant = tenant("friends", "500Gi"); + + reconciler.reconcile(tenant, null); + String firstToken = tokenValue(client.secrets() + .inNamespace("bluemap-friends") + .withName(PushTokenSecrets.SECRET_NAME) + .get()); + assertNotNull(firstToken, "token data must be present after the first reconcile"); + + // A second reconcile (the operator's regular resync, or any spec change) must not + // invalidate a token paper-worldpush may already be configured with. + reconciler.reconcile(tenant, null); + String secondToken = tokenValue(client.secrets() + .inNamespace("bluemap-friends") + .withName(PushTokenSecrets.SECRET_NAME) + .get()); + + assertEquals(firstToken, secondToken, "an already-provisioned push token must never be regenerated"); + } + + @Test + void refusesToAdoptAPushTokenSecretOwnedByAnotherTenant() { + // The namespace itself must already be correctly owned by this exact tenant (same UID), + // so the earlier namespace-ownership check does not fire first -- this test is only + // about the push-token secret's own, independent ownership check. + Tenant tenant = tenant("friends", "500Gi"); + client.namespaces() + .resource(new NamespaceBuilder() + .withNewMetadata() + .withName("bluemap-friends") + .withLabels(Map.of( + Labels.TENANT, + tenant.getMetadata().getName(), + Labels.TENANT_UID, + tenant.getMetadata().getUid())) + .endMetadata() + .build()) + .create(); + client.secrets() + .inNamespace("bluemap-friends") + .resource(new SecretBuilder() + .withNewMetadata() + .withName(PushTokenSecrets.SECRET_NAME) + .withNamespace("bluemap-friends") + .withLabels(Map.of( + Labels.TENANT, "friends", + Labels.TENANT_UID, UUID.randomUUID().toString())) + .endMetadata() + .withStringData(Map.of(PushTokenSecrets.TOKEN_KEY, "someone-elses-token")) + .build()) + .create(); + TenantReconciler reconciler = new TenantReconciler(client, OperatorConfig.defaults()); + + UpdateControl control = reconciler.reconcile(tenant, null); + + assertTrue(control.isPatchStatus()); + assertEquals(TenantReconciler.RESOURCE_CONFLICT_REASON, readyReason(tenant)); + } } diff --git a/paper-worldpush/build.gradle.kts b/paper-worldpush/build.gradle.kts new file mode 100644 index 0000000..dc072b3 --- /dev/null +++ b/paper-worldpush/build.gradle.kts @@ -0,0 +1,68 @@ +plugins { + alias(libs.plugins.shadow) +} + +dependencies { + // Paper API only, never paper-server/paper-mojangapi -- a plugin compiles against the API + // surface and runs inside whatever Paper build the operator actually deployed. See + // settings.gradle.kts for why this version is pinned independently of the rest of the catalog. + compileOnly(libs.paper.api) + + // AWS SDK v2 S3 client -- the same family already used by :ingest, so the project has exactly + // one S3 client/credential-provider chain instead of two. See settings.gradle.kts's comment + // on the `aws-sdk` version for the full rationale (also applies here unchanged). + // + // netty-nio-client excluded: it is the s3 artifact's *async*-client transport, pulled in as a + // direct dependency regardless of whether it is used. S3WorldUploader only ever makes + // blocking calls through the synchronous S3Client (apache5-client), so netty-nio-client's + // entire Netty dependency tree is dead weight here -- and, unlike in :ingest (a standalone + // process), shading an unrelated Netty version into a jar that loads inside a Paper server's + // own JVM (which already bundles Netty for its own networking) is a real classpath-collision + // risk worth avoiding outright rather than merely relocating. + implementation(platform(libs.aws.sdk.bom)) + implementation(libs.aws.sdk.s3) { + exclude(group = "software.amazon.awssdk", module = "netty-nio-client") + // slf4j-api excluded too, for the same reason: Paper already puts exactly one + // org.slf4j:slf4j-api on the server's runtime classpath (JavaPlugin#getSLF4JLogger() + // depends on it existing there), so shading a second, independently-versioned copy in + // alongside it is a classpath hazard rather than a safety net. compileOnly(libs.paper.api) + // already supplies the same API surface for compilation. + exclude(group = "org.slf4j", module = "slf4j-api") + } + + testImplementation(platform(libs.junit.bom)) + testImplementation(libs.junit.jupiter) + testRuntimeOnly(libs.junit.platform.launcher) +} + +tasks { + // paper-plugin.yml's `version: '${version}'` is a Gradle resource-filtering placeholder + // (Paper's own recommended pattern, see https://docs.papermc.io/paper/dev/project-setup/), + // not YAML/Paper syntax -- it must be expanded here or every plugin build reports the + // literal string "${version}" as its version. + processResources { + val props = mapOf("version" to project.version) + inputs.properties(props) + filesMatching("paper-plugin.yml") { + expand(props) + } + } + shadowJar { + archiveClassifier.set("") + archiveBaseName.set("apus-paper-worldpush") + // Fixed name instead of the default "apus-paper-worldpush-.jar" -- same + // rationale as telemetry-addon/build.gradle.kts and ingest/build.gradle.kts: whatever + // deploys this jar onto a Paper server (currently: a human, dropping it into `plugins/`) + // needs a stable file name to reference, not one that changes on every release-please bump. + archiveFileName.set("apus-paper-worldpush.jar") + // The AWS SDK is the only runtime dependency this plugin ships; Paper itself is + // compileOnly and provided by the server at runtime. Relocated to avoid clashing with + // any other plugin on the same server that also shades an AWS SDK, however unlikely. + relocate("software.amazon.awssdk", "net.onelitefeather.apus.paper.libs.awssdk") + relocate("org.reactivestreams", "net.onelitefeather.apus.paper.libs.reactivestreams") + relocate("org.apache.hc", "net.onelitefeather.apus.paper.libs.httpclient") + } + build { + dependsOn(shadowJar) + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitConfigSource.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitConfigSource.java new file mode 100644 index 0000000..bd3d318 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitConfigSource.java @@ -0,0 +1,40 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import org.bukkit.configuration.file.FileConfiguration; + +/** Adapts Bukkit's {@link FileConfiguration} (backing {@code config.yml}) to {@link ConfigSource}. */ +public final class BukkitConfigSource implements ConfigSource { + + private final FileConfiguration delegate; + + public BukkitConfigSource(FileConfiguration delegate) { + this.delegate = delegate; + } + + @Override + public String getString(String path) { + return delegate.getString(path); + } + + @Override + public long getLong(String path, long defaultValue) { + return delegate.getLong(path, defaultValue); + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitSaveCoordinator.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitSaveCoordinator.java new file mode 100644 index 0000000..a784d81 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/BukkitSaveCoordinator.java @@ -0,0 +1,109 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.logging.Level; +import java.util.logging.Logger; +import org.bukkit.World; +import org.bukkit.plugin.Plugin; + +/** + * {@link SaveCoordinator} backed by the real Bukkit/Paper world. Bridges from the calling thread + * (expected to be an async worker, never the main thread -- see below) onto Paper's {@code + * GlobalRegionScheduler} for each step, and blocks until that main-thread step has actually run, + * so callers can treat this as an ordinary synchronous dependency. + * + *

Must never be called from the main thread. Every method here submits work to the + * global region and then blocks waiting for it; calling this from the main thread would deadlock + * (the thread would be waiting for itself to become free). {@link WorldPushPlugin} only ever + * invokes {@link PushCycleRunner} from Paper's {@code AsyncScheduler}, never from a Bukkit + * event handler or a task scheduled on the main/region scheduler. + * + *

The world is resolved fresh on every call via {@code worldSupplier} rather than cached once, + * since a world can in principle be unloaded and reloaded while the plugin is running; if it is + * not currently loaded, each step is a silent no-op rather than an error -- there is nothing + * meaningful to save. + * + *

Untested. This class has no automated test coverage: exercising it needs a running + * Paper server (to observe a real world actually pause/resume autosave and be forced to disk), + * which is out of reach for this module's test suite -- see the phase 6 task report. + */ +public final class BukkitSaveCoordinator implements SaveCoordinator { + + private static final Logger LOGGER = Logger.getLogger(BukkitSaveCoordinator.class.getName()); + private static final long MAIN_THREAD_TIMEOUT_SECONDS = 30; + + private final Plugin plugin; + private final Supplier worldSupplier; + + public BukkitSaveCoordinator(Plugin plugin, Supplier worldSupplier) { + this.plugin = plugin; + this.worldSupplier = worldSupplier; + } + + @Override + public void disableAutoSave() { + runOnMainThreadAndAwait(world -> world.setAutoSave(false)); + } + + @Override + public void forceSave() { + runOnMainThreadAndAwait(World::save); + } + + @Override + public void enableAutoSave() { + runOnMainThreadAndAwait(world -> world.setAutoSave(true)); + } + + private void runOnMainThreadAndAwait(Consumer action) { + CompletableFuture future = new CompletableFuture<>(); + plugin.getServer().getGlobalRegionScheduler().run(plugin, task -> { + try { + World world = worldSupplier.get(); + if (world == null) { + LOGGER.warning("World is not currently loaded; skipping this save step."); + } else { + action.accept(world); + } + future.complete(null); + } catch (Throwable t) { + future.completeExceptionally(t); + } + }); + + try { + future.get(MAIN_THREAD_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Interrupted while waiting for a main-thread world save step.", e); + } catch (ExecutionException e) { + throw new IllegalStateException("Main-thread world save step failed.", e.getCause()); + } catch (TimeoutException e) { + LOGGER.log(Level.SEVERE, "Main-thread world save step did not complete within " + + MAIN_THREAD_TIMEOUT_SECONDS + "s.", e); + throw new IllegalStateException("Main-thread world save step timed out.", e); + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/ConfigSource.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/ConfigSource.java new file mode 100644 index 0000000..cbea87c --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/ConfigSource.java @@ -0,0 +1,35 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +/** + * The handful of config-file reads {@link WorldPushConfig} needs, kept deliberately narrow so + * tests can supply a plain in-memory implementation instead of a Bukkit {@code + * FileConfiguration} -- the same "depend on the smallest possible interface" pattern {@code + * ingest.S3Client} already uses for the same reason. + * + *

Paths are dotted, mirroring {@code config.yml}'s nesting (e.g. {@code "s3.access-key"}). + */ +public interface ConfigSource { + + /** Returns the string at {@code path}, or {@code null} if absent or not a string. */ + String getString(String path); + + /** Returns the long at {@code path}, or {@code defaultValue} if absent or not a number. */ + long getLong(String path, long defaultValue); +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyResult.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyResult.java new file mode 100644 index 0000000..45e2ff3 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyResult.java @@ -0,0 +1,35 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.util.List; + +/** + * The outcome of one {@link IncrementalWorldCopier#copyChanged} call. + * + * @param copiedRelativePaths the region files actually (re-)copied into staging this run, in the + * same relative-path form used as their eventual S3 key suffix + * @param copiedBytes total size of {@link #copiedRelativePaths} + * @param unchangedCount how many region files were considered but found unchanged + */ +public record CopyResult(List copiedRelativePaths, long copiedBytes, int unchangedCount) { + + public boolean isEmpty() { + return copiedRelativePaths.isEmpty(); + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyState.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyState.java new file mode 100644 index 0000000..6d6dd44 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/CopyState.java @@ -0,0 +1,130 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; +import java.util.logging.Level; +import java.util.logging.Logger; + +/** + * The persisted record of every region file {@link IncrementalWorldCopier} has seen, keyed by its + * relative path (e.g. {@code "world/region/r.0.0.mca"}). This is what makes a push cycle + * incremental across separate runs -- without it, every region file would look "new" every time. + * + *

Stored as a flat {@code key=size:mtime:checksum} properties file rather than a structured + * format: it is written and read only by this class, never by a human or another tool, so there + * is nothing a richer format would buy here that a one-line-per-entry format doesn't already + * give for free (human-readable, trivially diffable, zero extra dependency). + * + *

Crash safety. {@link #save(Path)} writes to a sibling temporary file and atomically + * renames it onto the target -- readers of the state file (the next push cycle) therefore only + * ever see either the previous complete state or the new complete state, never a half-written + * one. A crash before {@link #save(Path)} is called simply means the next cycle re-checksums + * (and, if actually changed, re-copies) a few files it did not strictly need to -- redundant + * work, never data loss or a corrupt state file. See {@link IncrementalWorldCopier} for the + * matching guarantee on the copy side. + */ +public final class CopyState { + + private static final Logger LOGGER = Logger.getLogger(CopyState.class.getName()); + + private final Map entries; + + private CopyState(Map entries) { + this.entries = entries; + } + + /** An empty state, as used before the very first push cycle. */ + public static CopyState empty() { + return new CopyState(new HashMap<>()); + } + + /** + * Loads state from {@code stateFile}. Returns an empty state (never throws) if the file does + * not exist yet, or if it exists but cannot be parsed -- a corrupt state file must never + * block push cycles from running; worst case it costs one fully non-incremental cycle. + */ + public static CopyState load(Path stateFile) { + if (!Files.isRegularFile(stateFile)) { + return empty(); + } + Properties properties = new Properties(); + try (InputStream in = Files.newInputStream(stateFile)) { + properties.load(in); + } catch (IOException e) { + LOGGER.log(Level.WARNING, "Could not read push state file " + stateFile + ", starting with empty state.", e); + return empty(); + } + Map entries = new HashMap<>(); + for (String key : properties.stringPropertyNames()) { + RegionFileState state = RegionFileState.decode(properties.getProperty(key)); + if (state != null) { + entries.put(key, state); + } + } + return new CopyState(entries); + } + + /** The recorded state for {@code relativePath}, or {@code null} if this file has never been seen. */ + public RegionFileState get(String relativePath) { + return entries.get(relativePath); + } + + /** Records (or replaces) the state for {@code relativePath}. */ + public void put(String relativePath, RegionFileState state) { + entries.put(relativePath, state); + } + + /** The number of region files currently tracked. */ + public int size() { + return entries.size(); + } + + /** + * Persists this state to {@code stateFile}, atomically. See the class Javadoc for the crash + * safety this provides. + */ + public void save(Path stateFile) throws IOException { + Path parent = stateFile.toAbsolutePath().getParent(); + if (parent != null) { + Files.createDirectories(parent); + } + Properties properties = new Properties(); + for (Map.Entry entry : entries.entrySet()) { + properties.setProperty(entry.getKey(), entry.getValue().encode()); + } + + Path tmp = Files.createTempFile(parent, stateFile.getFileName().toString(), ".tmp"); + try { + try (OutputStream out = Files.newOutputStream(tmp)) { + properties.store(out, "Apus world push -- region file copy state. Do not edit by hand."); + } + Files.move(tmp, stateFile, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE); + } finally { + Files.deleteIfExists(tmp); + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionLayout.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionLayout.java new file mode 100644 index 0000000..063e0c2 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionLayout.java @@ -0,0 +1,63 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; + +/** + * Locates the region-file directories for a world in the on-disk layout a Paper/Bukkit server + * actually uses: the primary world folder plus its {@code _nether}/{@code _the_end} siblings, + * each holding a {@code region} (or {@code DIM-1/region}, {@code DIM1/region}) folder. + * + *

This is deliberately the same {@code bukkit} layout {@code world-ingest}'s layout detector + * recognises server-side (design spec §6.2) -- picked so the staged copy needs no translation, + * not because this plugin re-implements that detector. A world that never generated the nether + * or the end simply has no matching directory; such dimensions are silently omitted rather than + * treated as an error. + */ +public final class DimensionLayout { + + private DimensionLayout() {} + + /** + * Returns the region directories that currently exist on disk for {@code worldName}, rooted + * at {@code serverRoot} (the directory Bukkit world folders live in -- typically the server's + * working directory). + */ + public static List forWorld(Path serverRoot, String worldName) { + List candidates = List.of( + new DimensionRegionDir(worldName + "/region", serverRoot.resolve(worldName).resolve("region")), + new DimensionRegionDir( + worldName + "_nether/DIM-1/region", + serverRoot.resolve(worldName + "_nether").resolve("DIM-1").resolve("region")), + new DimensionRegionDir( + worldName + "_the_end/DIM1/region", + serverRoot.resolve(worldName + "_the_end").resolve("DIM1").resolve("region"))); + + List existing = new ArrayList<>(); + for (DimensionRegionDir candidate : candidates) { + if (Files.isDirectory(candidate.sourceDir())) { + existing.add(candidate); + } + } + return existing; + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionRegionDir.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionRegionDir.java new file mode 100644 index 0000000..87f63f4 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/DimensionRegionDir.java @@ -0,0 +1,33 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.nio.file.Path; + +/** + * One dimension's region-file directory, on disk and as it will appear as a relative key both in + * the local staging directory and under the S3 staging prefix. + * + * @param relativePrefix the path from the world's parent directory to this dimension's {@code + * region} folder, using {@code /} separators regardless of platform (e.g. {@code + * "world_nether/DIM-1/region"}) -- this is intentionally identical to the on-disk Bukkit + * layout {@code world-ingest}'s layout detector already recognises (see the design spec, + * §6.2), so nothing needs translating on either end of the staging prefix + * @param sourceDir the actual directory on this server's disk to read {@code *.mca} files from + */ +public record DimensionRegionDir(String relativePrefix, Path sourceDir) {} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/HttpPushNotifier.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/HttpPushNotifier.java new file mode 100644 index 0000000..ce2aeb4 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/HttpPushNotifier.java @@ -0,0 +1,103 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.io.IOException; +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.time.Duration; + +/** + * Real {@link PushNotifier}, calling {@code POST {api-base-url}/api/push/{push-token}} (design + * spec §11.1) with the JDK's built-in {@link HttpClient} -- no need for a heavier HTTP dependency + * for one small POST call. + * + *

The push token is part of the request path, as the API contract in the design spec defines + * it. That means it is unavoidably present in the {@link URI} object built here -- but that URI + * is never logged, printed, or included in an exception message; only the host and a fixed, + * token-free label are. A future revision of the API that moves the token into an {@code + * Authorization} header instead would only need to change {@link #buildRequest}. + */ +public final class HttpPushNotifier implements PushNotifier { + + private static final Duration REQUEST_TIMEOUT = Duration.ofSeconds(30); + + private final HttpClient httpClient; + private final URI apiBaseUrl; + private final String pushToken; + + public HttpPushNotifier(URI apiBaseUrl, String pushToken) { + this.httpClient = HttpClient.newBuilder().connectTimeout(REQUEST_TIMEOUT).build(); + this.apiBaseUrl = apiBaseUrl; + this.pushToken = pushToken; + } + + @Override + public void notifyPushComplete(PushSummary summary) { + HttpRequest request = buildRequest(summary); + HttpResponse response; + try { + response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); + } catch (IOException e) { + throw new PushNotificationException( + "Could not reach the Apus API at " + apiBaseUrl.getHost() + " to report a completed push.", e); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new PushNotificationException("Interrupted while reporting a completed push.", e); + } + + if (response.statusCode() / 100 != 2) { + // Deliberately no response body in the message: it echoes request content back and + // this method has no way to know the API never includes the token in it. + throw new PushNotificationException("Apus API at " + apiBaseUrl.getHost() + + " rejected the push completion report with HTTP " + response.statusCode() + "."); + } + } + + private HttpRequest buildRequest(PushSummary summary) { + URI target = apiBaseUrl.resolve("/api/push/" + pushToken); + // Exactly the two fields PushReportRequest (module api, package + // net.onelitefeather.apus.api.rest.push) deserializes -- see PushSummary's Javadoc for + // why the rest of the summary never goes on the wire. + String body = "{\"sourceName\":\"" + jsonEscape(summary.sourceName()) + "\",\"version\":\"" + + jsonEscape(summary.version()) + "\"}"; + return HttpRequest.newBuilder(target) + .timeout(REQUEST_TIMEOUT) + .header("Content-Type", "application/json") + .POST(HttpRequest.BodyPublishers.ofString(body)) + .build(); + } + + private static String jsonEscape(String value) { + return value.replace("\\", "\\\\").replace("\"", "\\\""); + } + + /** Thrown when the completion report could not be delivered or was rejected by the API. */ + public static final class PushNotificationException extends RuntimeException { + + PushNotificationException(String message) { + super(message); + } + + PushNotificationException(String message, Throwable cause) { + super(message, cause); + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/IncrementalWorldCopier.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/IncrementalWorldCopier.java new file mode 100644 index 0000000..6018620 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/IncrementalWorldCopier.java @@ -0,0 +1,162 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.DirectoryStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.security.DigestInputStream; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.HexFormat; +import java.util.List; + +/** + * Copies the region files of a running world into a local staging directory, touching only files + * that actually changed since the last run. + * + *

Why incremental at all. Copying an entire world's region files on every push cycle is + * the safe default but does not scale: for a large world it can take long enough that keeping the + * server's autosave paused for the whole copy (see {@link SaveCoordinator}) would make the server + * noticeably stutter. Restricting the copy to changed files keeps the disruptive part -- the + * brief autosave pause plus one forced save -- short regardless of world size; only the copy + * itself scales with how much actually changed, and that step runs off the main thread anyway. + * + *

What "changed" means. Two signals, used together, exactly as the design brief + * specifies ("nur Region-Dateien mit geänderter Änderungszeit oder Prüfsumme"): + * + *

    + *
  1. {@code size}/{@code mtime} is the cheap first check (a single {@code stat}, no file + * content is read). If both match the last recorded {@link RegionFileState}, the file is + * skipped outright -- this is what keeps a cycle over an untouched world near-instant. + *
  2. If either differs, the file's SHA-256 checksum is computed and compared against the last + * recorded one. Only a genuine checksum mismatch causes an actual copy; a file whose mtime + * moved but whose bytes did not (a `touch`, a filesystem quirk, a region file rewritten with + * identical content) updates its recorded {@code size}/{@code mtime} for next time without + * being re-uploaded. + *
+ * + *

Crash safety. Each file is copied to a temporary file in the same destination + * directory (guaranteeing the same filesystem) and then moved onto its final name with {@link + * StandardCopyOption#ATOMIC_MOVE}. A reader of the staging directory therefore only ever sees a + * region file as either fully absent or fully present at its final size -- never truncated or + * half-written. If the copy of one file fails, its temporary file is deleted and the exception + * propagates; files already moved into place before the failure stay in place (there is no + * transactional rollback across the whole batch -- and none is needed, since each file's own + * atomicity is what matters: a caller that retries the whole cycle later will simply find those + * files unchanged next time via the check above). {@link #copyChanged} does not persist {@code + * state} itself -- see {@link CopyState}'s own Javadoc for why leaving that to the caller, once + * per successful cycle, is what makes the persisted state crash-safe too. + */ +public final class IncrementalWorldCopier { + + private static final String DIGEST_ALGORITHM = "SHA-256"; + private static final String REGION_FILE_SUFFIX = ".mca"; + + /** + * Copies every changed {@code *.mca} file from {@code regionDirs} into {@code stagingRoot}, + * updating {@code state} in place for every file examined (changed or not). Does not persist + * {@code state}; the caller does that once the whole cycle -- across all dimensions -- has + * completed successfully. + * + * @throws IOException if listing a region directory or copying a file fails; already-copied + * files from earlier in this call remain in {@code stagingRoot} (see the class Javadoc) + */ + public CopyResult copyChanged(List regionDirs, Path stagingRoot, CopyState state) + throws IOException { + List copied = new ArrayList<>(); + long copiedBytes = 0; + int unchanged = 0; + + for (DimensionRegionDir dimension : regionDirs) { + for (Path sourceFile : listRegionFiles(dimension.sourceDir())) { + String relativePath = dimension.relativePrefix() + "/" + sourceFile.getFileName(); + long size = Files.size(sourceFile); + long lastModifiedMillis = Files.getLastModifiedTime(sourceFile).toMillis(); + + RegionFileState prior = state.get(relativePath); + if (prior != null && prior.size() == size && prior.lastModifiedMillis() == lastModifiedMillis) { + unchanged++; + continue; + } + + String checksum = sha256(sourceFile); + if (prior != null && prior.checksum().equals(checksum)) { + // Same content, only the timestamp moved -- refresh the cheap-check fields + // so the next cycle takes the fast path again, but do not copy or upload. + state.put(relativePath, new RegionFileState(size, lastModifiedMillis, checksum)); + unchanged++; + continue; + } + + copyAtomically(sourceFile, stagingRoot.resolve(relativePath)); + state.put(relativePath, new RegionFileState(size, lastModifiedMillis, checksum)); + copied.add(relativePath); + copiedBytes += size; + } + } + + return new CopyResult(List.copyOf(copied), copiedBytes, unchanged); + } + + private static List listRegionFiles(Path regionDir) throws IOException { + List files = new ArrayList<>(); + try (DirectoryStream stream = + Files.newDirectoryStream(regionDir, entry -> Files.isRegularFile(entry) + && entry.getFileName().toString().endsWith(REGION_FILE_SUFFIX))) { + for (Path entry : stream) { + files.add(entry); + } + } + files.sort(Path::compareTo); + return files; + } + + private static void copyAtomically(Path sourceFile, Path destination) throws IOException { + Path destinationDir = destination.toAbsolutePath().getParent(); + Files.createDirectories(destinationDir); + Path tmp = Files.createTempFile(destinationDir, destination.getFileName().toString(), ".tmp"); + try { + Files.copy(sourceFile, tmp, StandardCopyOption.REPLACE_EXISTING); + Files.move(tmp, destination, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + } finally { + Files.deleteIfExists(tmp); + } + } + + private static String sha256(Path file) throws IOException { + MessageDigest digest; + try { + digest = MessageDigest.getInstance(DIGEST_ALGORITHM); + } catch (NoSuchAlgorithmException e) { + // Every JDK ships SHA-256; see java.security.MessageDigest's own guarantee. + throw new IllegalStateException(e); + } + byte[] buffer = new byte[8192]; + try (InputStream in = new DigestInputStream(Files.newInputStream(file), digest)) { + while (in.read(buffer) != -1) { + // DigestInputStream updates the digest as a side effect of reading. + } + } + return HexFormat.of().formatHex(digest.digest()); + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushCycleRunner.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushCycleRunner.java new file mode 100644 index 0000000..53723bf --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushCycleRunner.java @@ -0,0 +1,155 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.io.IOException; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.util.List; +import java.util.logging.Logger; + +/** + * Runs one complete push cycle: quiesce autosave, force a save, copy changed region files, + * upload them, and report completion -- steps 1 through 3 of the design brief, in order. + * + *

Threading contract. This class itself does not touch any thread -- it just calls + * {@link SaveCoordinator}, {@link IncrementalWorldCopier}, {@link WorldUploader} and {@link + * PushNotifier} in sequence. The threading guarantee the design brief asks for ("nothing may + * block the server thread") comes entirely from which thread {@link #runCycle()} itself is + * called on and which {@link SaveCoordinator} implementation it is given: {@link + * WorldPushPlugin} only ever calls this from Paper's {@code AsyncScheduler}, and {@link + * BukkitSaveCoordinator} bridges only its own three method calls onto the main thread, blocking + * this (async) thread while it waits -- never the other way around. + * + *

State progression. {@link CopyState} is only persisted to disk once the entire + * cycle -- copy, upload, and notify -- has succeeded. If anything after the copy step fails + * (an upload, the notification), the in-memory state mutations {@link IncrementalWorldCopier} + * made are simply discarded; the next cycle reloads the last good state from disk and will + * see the same files as changed again. See {@link CopyState} and {@link IncrementalWorldCopier} + * for the matching per-file crash-safety guarantee this builds on. + */ +public final class PushCycleRunner { + + private static final Logger LOGGER = Logger.getLogger(PushCycleRunner.class.getName()); + + /** + * Formats each push cycle's {@code version} identifier -- mirrors the timestamp-style version + * ids the rest of Apus already uses for source versions (e.g. {@code + * S3SourceConnector}'s {@code 2026-08-01T00-00-00Z.zip} object keys), minus a file extension + * since a push cycle uploads many individual region files rather than one archive. + */ + private static final DateTimeFormatter VERSION_FORMAT = + DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH-mm-ss'Z'").withZone(ZoneOffset.UTC); + + private final IncrementalWorldCopier copier; + private final SaveCoordinator saveCoordinator; + private final WorldUploader uploader; + private final PushNotifier notifier; + private final Path serverRoot; + private final Path stagingRoot; + private final Path stateFile; + private final WorldPushConfig config; + private final Clock clock; + + public PushCycleRunner( + IncrementalWorldCopier copier, + SaveCoordinator saveCoordinator, + WorldUploader uploader, + PushNotifier notifier, + Path serverRoot, + Path stagingRoot, + Path stateFile, + WorldPushConfig config) { + this(copier, saveCoordinator, uploader, notifier, serverRoot, stagingRoot, stateFile, config, Clock.systemUTC()); + } + + /** Same as the public constructor, but with an injectable {@link Clock} -- for deterministic version-id tests. */ + PushCycleRunner( + IncrementalWorldCopier copier, + SaveCoordinator saveCoordinator, + WorldUploader uploader, + PushNotifier notifier, + Path serverRoot, + Path stagingRoot, + Path stateFile, + WorldPushConfig config, + Clock clock) { + this.copier = copier; + this.saveCoordinator = saveCoordinator; + this.uploader = uploader; + this.notifier = notifier; + this.serverRoot = serverRoot; + this.stagingRoot = stagingRoot; + this.stateFile = stateFile; + this.config = config; + this.clock = clock; + } + + /** + * Runs one push cycle to completion. Must be called off the main thread -- see the class + * Javadoc. + * + * @throws IOException if reading region directories or copying/persisting state fails + * @throws HttpPushNotifier.PushNotificationException if the completion report is rejected or + * unreachable (only thrown by the real {@link PushNotifier}; a test fake may throw + * whatever it likes) + */ + public void runCycle() throws IOException { + saveCoordinator.disableAutoSave(); + try { + saveCoordinator.forceSave(); + } finally { + // Always re-enabled, even if forceSave() failed -- a server permanently stuck + // without autosave because one push cycle had a bad day is a worse outcome than + // that cycle's copy being skipped or stale. + saveCoordinator.enableAutoSave(); + } + + CopyState state = CopyState.load(stateFile); + List regionDirs = DimensionLayout.forWorld(serverRoot, config.worldName()); + CopyResult result = copier.copyChanged(regionDirs, stagingRoot, state); + + if (result.isEmpty()) { + // Still persisted: copyChanged() may have refreshed size/mtime for files whose + // content did not actually change (see IncrementalWorldCopier), and there is + // nothing risky about saving that -- no upload or notification happened. + state.save(stateFile); + LOGGER.fine("Push cycle: no region files changed, nothing to upload."); + return; + } + + for (String relativePath : result.copiedRelativePaths()) { + uploader.upload(stagingRoot.resolve(relativePath), config.s3StagingPrefix() + relativePath); + } + + String version = VERSION_FORMAT.format(Instant.now(clock)); + notifier.notifyPushComplete(new PushSummary( + config.sourceName(), + version, + config.worldName(), + result.copiedRelativePaths().size(), + result.copiedBytes())); + + state.save(stateFile); + LOGGER.info("Push cycle: uploaded " + result.copiedRelativePaths().size() + " region file(s), " + + result.copiedBytes() + " bytes."); + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushNotifier.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushNotifier.java new file mode 100644 index 0000000..539c520 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushNotifier.java @@ -0,0 +1,34 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +/** + * Reports a completed push cycle to the Apus API ({@code POST /api/push/{token}}, design spec + * §11.1), so the {@code push} ingest connector knows a new version is waiting in the staging + * prefix. Kept as its own interface so {@link PushCycleRunner} can be tested with a fake instead + * of a real HTTP call -- see {@link HttpPushNotifier} for the real implementation. + */ +public interface PushNotifier { + + /** + * Reports {@code summary} as a completed push. Implementations are expected to throw on + * failure (network error, non-2xx response) rather than swallow it -- {@link PushCycleRunner} + * relies on that to decide whether the cycle's state may be persisted as "done". + */ + void notifyPushComplete(PushSummary summary); +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushSummary.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushSummary.java new file mode 100644 index 0000000..fc41791 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/PushSummary.java @@ -0,0 +1,39 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +/** + * What one completed push cycle reports to the Apus API, so {@code POST /api/push/{token}} has + * enough context to create a {@code WorldIngest} plus log/display without a round-trip back to + * this server. + * + *

{@code sourceName} and {@code version} are exactly the two fields {@code + * PushReportRequest} (module {@code api}, package {@code + * net.onelitefeather.apus.api.rest.push}) deserializes the request body into -- {@link + * HttpPushNotifier} sends only those two on the wire. {@code worldName}/{@code fileCount}/{@code + * bytesUploaded} are not part of that contract (the API already knows the world name from the + * target {@code WorldSource}'s own configured worlds, and file/byte counts are this plugin's own + * telemetry, not the API's concern); they stay on this record purely so a {@link PushNotifier} + * implementation can log/display them locally without a second parameter list. + * + * @param sourceName the target {@code push}-type {@code WorldSource}'s name, from {@code + * WorldPushConfig#sourceName()} -- becomes {@code PushReportRequest.sourceName()} + * @param version this push cycle's identifier -- becomes {@code PushReportRequest.version()} and, + * from there, {@code WorldIngest.spec.sourceVersion} + */ +public record PushSummary(String sourceName, String version, String worldName, int fileCount, long bytesUploaded) {} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/RegionFileState.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/RegionFileState.java new file mode 100644 index 0000000..d4c8f4a --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/RegionFileState.java @@ -0,0 +1,47 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +/** + * The last known state of one region file, as recorded after it was last copied into staging. + * + *

{@code size}/{@code lastModifiedMillis} are the cheap "did the OS report a change" check; + * {@code checksum} (SHA-256, hex-encoded) is only computed when that cheap check trips, to + * confirm the content actually differs -- see {@link IncrementalWorldCopier}'s class Javadoc for + * why both signals are used together. + */ +public record RegionFileState(long size, long lastModifiedMillis, String checksum) { + + /** Serialises to the single-line format {@link CopyState} persists, e.g. {@code "1024:1700000000000:ab12..."}. */ + String encode() { + return size + ":" + lastModifiedMillis + ":" + checksum; + } + + /** Parses the format {@link #encode()} produces; returns {@code null} if {@code line} is malformed. */ + static RegionFileState decode(String line) { + String[] parts = line.split(":", 3); + if (parts.length != 3) { + return null; + } + try { + return new RegionFileState(Long.parseLong(parts[0]), Long.parseLong(parts[1]), parts[2]); + } catch (NumberFormatException e) { + return null; + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/S3WorldUploader.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/S3WorldUploader.java new file mode 100644 index 0000000..8221e6d --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/S3WorldUploader.java @@ -0,0 +1,70 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.net.URI; +import java.nio.file.Path; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; + +/** + * Real {@link WorldUploader}, writing region files into the tenant's staging prefix via the AWS + * SDK v2 S3 client (the same client family {@code ingest.S3Client} wraps, path-style access so + * this also works unchanged against Rook/Ceph -- see {@code settings.gradle.kts}'s {@code + * aws-sdk} version comment for the shared rationale). + * + *

Never logs {@link WorldPushConfig#s3AccessKey()}/{@link WorldPushConfig#s3SecretKey()}; they + * are only ever passed into {@link StaticCredentialsProvider}, never rendered to a string. + */ +public final class S3WorldUploader implements WorldUploader, AutoCloseable { + + private final S3Client delegate; + private final String bucket; + + private S3WorldUploader(S3Client delegate, String bucket) { + this.delegate = delegate; + this.bucket = bucket; + } + + /** Builds a real uploader from {@code config}'s S3 settings. */ + public static S3WorldUploader create(WorldPushConfig config) { + S3Client client = S3Client.builder() + .region(Region.of(config.s3Region())) + .endpointOverride(URI.create(config.s3Endpoint())) + .credentialsProvider(StaticCredentialsProvider.create( + AwsBasicCredentials.create(config.s3AccessKey(), config.s3SecretKey()))) + .forcePathStyle(true) + .build(); + return new S3WorldUploader(client, config.s3Bucket()); + } + + @Override + public void upload(Path localFile, String s3Key) { + delegate.putObject( + PutObjectRequest.builder().bucket(bucket).key(s3Key).build(), + software.amazon.awssdk.core.sync.RequestBody.fromFile(localFile)); + } + + @Override + public void close() { + delegate.close(); + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/SaveCoordinator.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/SaveCoordinator.java new file mode 100644 index 0000000..fa70cdc --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/SaveCoordinator.java @@ -0,0 +1,43 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +/** + * The three save-related steps a push cycle needs from the live world, kept as their own + * interface so {@link PushCycleRunner} can be exercised without a running Paper server -- the + * copy logic itself is Bukkit-free, but the "pause autosave, force one save, resume autosave" + * sequence around it is inherently Bukkit-API-shaped, and PaperMC's guidance is that all such + * calls belong on the main thread. See {@link BukkitSaveCoordinator} for the real, main-thread + * bridging implementation, and the phase 6 task report for why that implementation itself has no + * automated test -- it needs a live server. + * + *

Every method is expected to block the calling thread until the underlying main-thread step + * has actually completed (not merely been scheduled), so that {@link PushCycleRunner} can treat + * this as a simple synchronous sequence despite the work happening on a different thread. + */ +public interface SaveCoordinator { + + /** Disables the world's automatic periodic saving, so it cannot race the forced save below. */ + void disableAutoSave(); + + /** Forces one synchronous save of the world's current state to disk. */ + void forceSave(); + + /** Re-enables automatic periodic saving. Always called, even if {@link #forceSave()} failed. */ + void enableAutoSave(); +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushConfig.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushConfig.java new file mode 100644 index 0000000..c22e62c --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushConfig.java @@ -0,0 +1,228 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.net.URI; + +/** + * The plugin's complete configuration, read from {@code config.yml} (via {@link ConfigSource}) + * and validated eagerly. + * + *

{@link #from(ConfigSource)} is the single place every required key is checked. It either + * returns a fully valid configuration or throws {@link ConfigurationException} before any push + * cycle is scheduled -- mirroring {@code ingest.IngestConfig}'s "fail before anything runs" + * contract in the sibling module. + * + *

Two distinct credentials are held here, deliberately not conflated (see {@code config.yml}'s + * comments for the full rationale): + * + *

    + *
  • {@link #s3AccessKey()}/{@link #s3SecretKey()} -- tenant-scoped S3 bucket credentials, + * used to write region files directly into the staging prefix. + *
  • {@link #pushToken()} -- a narrow, tenant-bound {@code world:push} service token (§10.3 of + * the design spec), used only to authenticate the completion report to the Apus API. + *
+ */ +public final class WorldPushConfig { + + private final String worldName; + private final String tenant; + private final String sourceName; + private final String pushToken; + private final String stagingDirectory; + private final String s3Endpoint; + private final String s3Bucket; + private final String s3Region; + private final String s3AccessKey; + private final String s3SecretKey; + private final String s3StagingPrefix; + private final URI apusApiBaseUrl; + private final long intervalMinutes; + + private WorldPushConfig( + String worldName, + String tenant, + String sourceName, + String pushToken, + String stagingDirectory, + String s3Endpoint, + String s3Bucket, + String s3Region, + String s3AccessKey, + String s3SecretKey, + String s3StagingPrefix, + URI apusApiBaseUrl, + long intervalMinutes) { + this.worldName = worldName; + this.tenant = tenant; + this.sourceName = sourceName; + this.pushToken = pushToken; + this.stagingDirectory = stagingDirectory; + this.s3Endpoint = s3Endpoint; + this.s3Bucket = s3Bucket; + this.s3Region = s3Region; + this.s3AccessKey = s3AccessKey; + this.s3SecretKey = s3SecretKey; + this.s3StagingPrefix = s3StagingPrefix; + this.apusApiBaseUrl = apusApiBaseUrl; + this.intervalMinutes = intervalMinutes; + } + + /** + * Reads and validates every configuration value the plugin needs from {@code source}. + * + * @throws ConfigurationException if a required key is missing/blank, or a value cannot be + * parsed (e.g. {@code apus.api-base-url} is not a valid URI) + */ + public static WorldPushConfig from(ConfigSource source) { + String worldName = requireNonBlank(source, "world-name"); + String tenant = requireNonBlank(source, "tenant"); + String sourceName = requireNonBlank(source, "world-source-name"); + String pushToken = requireNonBlank(source, "push-token"); + String stagingDirectory = orDefault(source.getString("staging-directory"), "apus-worldpush-staging"); + + String s3Endpoint = requireNonBlank(source, "s3.endpoint"); + String s3Bucket = requireNonBlank(source, "s3.bucket"); + String s3Region = orDefault(source.getString("s3.region"), "us-east-1"); + String s3AccessKey = requireNonBlank(source, "s3.access-key"); + String s3SecretKey = requireNonBlank(source, "s3.secret-key"); + String s3StagingPrefix = normalizePrefix(orDefault(source.getString("s3.staging-prefix"), "staging/")); + + String apiBaseUrlRaw = requireNonBlank(source, "apus.api-base-url"); + URI apusApiBaseUrl; + try { + apusApiBaseUrl = URI.create(apiBaseUrlRaw); + } catch (IllegalArgumentException e) { + throw new ConfigurationException("apus.api-base-url is not a valid URI: '" + apiBaseUrlRaw + "'"); + } + if (apusApiBaseUrl.getScheme() == null || apusApiBaseUrl.getHost() == null) { + throw new ConfigurationException( + "apus.api-base-url must be an absolute URL with a scheme and host, got: '" + apiBaseUrlRaw + "'"); + } + + long intervalMinutes = source.getLong("schedule.interval-minutes", 30); + if (intervalMinutes <= 0) { + throw new ConfigurationException("schedule.interval-minutes must be a positive integer, got: " + intervalMinutes); + } + + return new WorldPushConfig( + worldName, + tenant, + sourceName, + pushToken, + stagingDirectory, + s3Endpoint, + s3Bucket, + s3Region, + s3AccessKey, + s3SecretKey, + s3StagingPrefix, + apusApiBaseUrl, + intervalMinutes); + } + + private static String normalizePrefix(String prefix) { + String trimmed = prefix.startsWith("/") ? prefix.substring(1) : prefix; + return trimmed.endsWith("/") || trimmed.isEmpty() ? trimmed : trimmed + "/"; + } + + private static String orDefault(String value, String defaultValue) { + return (value == null || value.isBlank()) ? defaultValue : value; + } + + private static String requireNonBlank(ConfigSource source, String path) { + String value = source.getString(path); + if (value == null || value.isBlank()) { + throw new ConfigurationException(path + " is required but was not set in config.yml."); + } + return value; + } + + public String worldName() { + return worldName; + } + + public String tenant() { + return tenant; + } + + /** + * The target {@code push}-type {@code WorldSource}'s name, within {@link #tenant()}'s + * namespace -- becomes {@code PushReportRequest.sourceName()} in every completion report (see + * {@link PushSummary}). Distinct from {@link #tenant()}: the namespace a push token + * authorizes is resolved from the token alone (design spec §10.3), but a tenant may run more + * than one push-type source, so the token by itself is not enough to pick which one this + * server's uploads belong to. + */ + public String sourceName() { + return sourceName; + } + + /** The narrowly-scoped {@code world:push} service token -- never log this value. */ + public String pushToken() { + return pushToken; + } + + /** Relative or absolute path to the local staging directory; resolved against the plugin's data folder. */ + public String stagingDirectory() { + return stagingDirectory; + } + + public String s3Endpoint() { + return s3Endpoint; + } + + public String s3Bucket() { + return s3Bucket; + } + + public String s3Region() { + return s3Region; + } + + /** Tenant-scoped S3 access key -- never log this value. */ + public String s3AccessKey() { + return s3AccessKey; + } + + /** Tenant-scoped S3 secret key -- never log this value. */ + public String s3SecretKey() { + return s3SecretKey; + } + + /** Key prefix within {@link #s3Bucket()} that staged region files are uploaded under, always ending in {@code "/"}. */ + public String s3StagingPrefix() { + return s3StagingPrefix; + } + + public URI apusApiBaseUrl() { + return apusApiBaseUrl; + } + + public long intervalMinutes() { + return intervalMinutes; + } + + /** Thrown when {@code config.yml} is missing a required key or holds an invalid value. */ + public static final class ConfigurationException extends RuntimeException { + + ConfigurationException(String message) { + super(message); + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushPlugin.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushPlugin.java new file mode 100644 index 0000000..bf72c9e --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldPushPlugin.java @@ -0,0 +1,117 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.io.IOException; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import org.bukkit.plugin.java.JavaPlugin; + +/** + * Entry point of the Apus world-push plugin: periodically takes a consistent, incremental copy + * of this server's configured world and uploads it to a staging prefix in S3, then reports the + * new version to the Apus API. See the design spec, §3 ("Push: async + inkrementell in + * Staging-Prefix") and §6.4. + * + *

Every push cycle runs on Paper's {@code AsyncScheduler} -- never the main thread, never + * {@code BukkitScheduler#runTaskAsynchronously} (Paper's own guidance prefers the newer + * schedulers, see {@code docs.papermc.io/paper/dev/folia-support}). The only main-thread work is + * the brief autosave-pause-and-force-save step inside {@link BukkitSaveCoordinator}, bridged back + * onto the main thread per call. A cycle still running when the next one would start is skipped + * rather than queued or run concurrently -- {@link #cycleRunning} enforces that. + */ +public final class WorldPushPlugin extends JavaPlugin { + + private final AtomicBoolean cycleRunning = new AtomicBoolean(false); + + private WorldPushConfig config; + private S3WorldUploader uploader; + + @Override + public void onEnable() { + saveDefaultConfig(); + reloadConfig(); + + try { + config = WorldPushConfig.from(new BukkitConfigSource(getConfig())); + } catch (WorldPushConfig.ConfigurationException e) { + getSLF4JLogger().error("Invalid config.yml, disabling: {}", e.getMessage()); + getServer().getPluginManager().disablePlugin(this); + return; + } + + uploader = S3WorldUploader.create(config); + + Path serverRoot = getServer().getWorldContainer().toPath(); + Path stagingRoot = getDataFolder().toPath().resolve(config.stagingDirectory()); + Path stateFile = getDataFolder().toPath().resolve("push-state.properties"); + + SaveCoordinator saveCoordinator = + new BukkitSaveCoordinator(this, () -> getServer().getWorld(config.worldName())); + PushNotifier notifier = new HttpPushNotifier(config.apusApiBaseUrl(), config.pushToken()); + PushCycleRunner runner = new PushCycleRunner( + new IncrementalWorldCopier(), saveCoordinator, uploader, notifier, serverRoot, stagingRoot, stateFile, + config); + + getServer() + .getAsyncScheduler() + .runAtFixedRate( + this, + task -> runCycleGuarded(runner), + config.intervalMinutes(), + config.intervalMinutes(), + TimeUnit.MINUTES); + + getSLF4JLogger() + .info( + "Apus world push enabled for world '{}', tenant '{}', source '{}', every {} minute(s).", + config.worldName(), + config.tenant(), + config.sourceName(), + config.intervalMinutes()); + } + + @Override + public void onDisable() { + getServer().getAsyncScheduler().cancelTasks(this); + getServer().getGlobalRegionScheduler().cancelTasks(this); + if (uploader != null) { + uploader.close(); + } + } + + private void runCycleGuarded(PushCycleRunner runner) { + if (!cycleRunning.compareAndSet(false, true)) { + getSLF4JLogger().debug("Skipping this push cycle: the previous one is still running."); + return; + } + try { + runner.runCycle(); + } catch (IOException e) { + getSLF4JLogger().warn("Push cycle failed: {}", e.getMessage(), e); + } catch (RuntimeException e) { + // Covers HttpPushNotifier.PushNotificationException and any unexpected failure from + // the uploader/save coordinator -- a bad cycle must never crash the scheduler and + // silently stop all future pushes. + getSLF4JLogger().warn("Push cycle failed: {}", e.getMessage(), e); + } finally { + cycleRunning.set(false); + } + } +} diff --git a/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldUploader.java b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldUploader.java new file mode 100644 index 0000000..7930516 --- /dev/null +++ b/paper-worldpush/src/main/java/net/onelitefeather/apus/paper/WorldUploader.java @@ -0,0 +1,32 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import java.nio.file.Path; + +/** + * The one upload operation {@link PushCycleRunner} needs, kept deliberately narrow -- mirrors + * {@code ingest.S3Client}'s reasoning in the sibling module: tests substitute an in-memory fake + * instead of talking to real S3-compatible storage, and the real implementation ({@link + * S3WorldUploader}) is the only place that knows about the AWS SDK. + */ +public interface WorldUploader { + + /** Uploads {@code localFile}'s current contents to {@code s3Key}, overwriting any object already there. */ + void upload(Path localFile, String s3Key); +} diff --git a/paper-worldpush/src/main/resources/config.yml b/paper-worldpush/src/main/resources/config.yml new file mode 100644 index 0000000..a08f6de --- /dev/null +++ b/paper-worldpush/src/main/resources/config.yml @@ -0,0 +1,62 @@ +# Apus World Push -- configuration +# +# Copied out to plugins/ApusWorldPush/config.yml on first start. Fill in the values below +# before enabling push cycles; the plugin fails fast at startup (nothing is scheduled) if a +# required value is left blank. +# +# Credentials in this file (push-token, s3.access-key, s3.secret-key) are never written to +# the server log or console. Keep this file's permissions restricted the same way you would +# for server.properties. + +# The Bukkit world (folder name) to push, e.g. "world". Its Bukkit-layout siblings +# ("_nether", "_the_end") are picked up automatically if present -- the same +# layout world-ingest already recognises server-side (see docs/superpowers/specs, §6.2). +world-name: world + +# The Apus tenant this server belongs to. Informational only (shown in this plugin's own log +# lines) -- it grants no access by itself and is never sent to the Apus API. The push-token +# below is what actually authorizes anything, and world-source-name below is what selects +# which of the tenant's WorldSource resources this server's uploads are reported against. +tenant: '' + +# The name of the "push"-type WorldSource resource (in the Apus UI/API) this server's uploads +# belong to. A tenant may run more than one push source, so the push-token alone (tenant-bound, +# not source-bound) is not enough to pick one -- this is. Sent as "sourceName" in every +# completion report to POST /api/push/{push-token}. +world-source-name: '' + +# A tenant-bound, narrowly-scoped ("world:push") service token -- see §10.3 of the Apus +# design spec. Deliberately not a user login: a person leaving the team must never be able +# to take this server's uploads down with them. Ask a tenant-owner in the Apus UI/API to +# mint one. Sent as a path segment (POST /api/push/{push-token}), never as a header and +# never logged. +push-token: '' + +# Where the staged copy is written before/while it is uploaded, on this server's own disk. +# Must have room for roughly one incremental push's worth of region files, not the whole +# world -- see IncrementalWorldCopier. +staging-directory: apus-worldpush-staging + +s3: + endpoint: '' + bucket: '' + region: us-east-1 + # Direct, tenant-scoped bucket credentials (the same kind of per-tenant CephObjectStoreUser + # credentials the rest of Apus already uses, see §10.1/§10.2 of the design spec) -- distinct + # from push-token above, which authenticates the completion report to the Apus API, not the + # S3 write itself. + access-key: '' + secret-key: '' + # Region files are written under this prefix, never directly into the tenant's normal + # bundle path -- see §6.4 of the design spec ("Push-Quellen ... Staging-Prefix"). + staging-prefix: staging/ + +apus: + # Base URL of the Apus API, e.g. https://apus.example.org. POST {api-base-url}/api/push/{push-token} + # is called once a push cycle's uploads have all succeeded. + api-base-url: '' + +schedule: + # How often a push cycle runs, in minutes. A cycle that is still running when the next one + # would start is skipped, not queued -- see WorldPushPlugin. + interval-minutes: 30 diff --git a/paper-worldpush/src/main/resources/paper-plugin.yml b/paper-worldpush/src/main/resources/paper-plugin.yml new file mode 100644 index 0000000..cb75a2f --- /dev/null +++ b/paper-worldpush/src/main/resources/paper-plugin.yml @@ -0,0 +1,13 @@ +name: ApusWorldPush +version: '${version}' +main: net.onelitefeather.apus.paper.WorldPushPlugin +description: Pushes an incremental, consistent copy of this server's world to Apus. +author: OneLiteFeather +website: https://github.com/OneLiteFeatherNET/Apus +api-version: '26.2' +# This plugin schedules its own work through Paper's AsyncScheduler/GlobalRegionScheduler +# instead of BukkitScheduler, and touches exactly one World per configured push job -- see +# WorldPushPlugin's class Javadoc. It has not been exercised against a live Folia server +# (see the phase 6 task report), so this is left unset (defaults to false/unsupported) +# rather than claimed without verification. +# folia-supported: false diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/CopyStateTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/CopyStateTest.java new file mode 100644 index 0000000..334f28d --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/CopyStateTest.java @@ -0,0 +1,101 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class CopyStateTest { + + @TempDir + Path tmp; + + @Test + void loadingAMissingFileReturnsEmptyState() { + CopyState state = CopyState.load(tmp.resolve("does-not-exist.properties")); + + assertEquals(0, state.size()); + assertNull(state.get("world/region/r.0.0.mca")); + } + + @Test + void savedStateRoundTripsThroughLoad() throws IOException { + Path stateFile = tmp.resolve("push-state.properties"); + CopyState original = CopyState.empty(); + original.put("world/region/r.0.0.mca", new RegionFileState(1024, 1_700_000_000_000L, "abc123")); + original.put("world_nether/DIM-1/region/r.-1.0.mca", new RegionFileState(2048, 1_700_000_001_000L, "def456")); + + original.save(stateFile); + CopyState reloaded = CopyState.load(stateFile); + + assertEquals(2, reloaded.size()); + assertEquals(new RegionFileState(1024, 1_700_000_000_000L, "abc123"), reloaded.get("world/region/r.0.0.mca")); + assertEquals( + new RegionFileState(2048, 1_700_000_001_000L, "def456"), + reloaded.get("world_nether/DIM-1/region/r.-1.0.mca")); + } + + @Test + void aCorruptStateFileFallsBackToEmptyInsteadOfThrowing() throws IOException { + Path stateFile = tmp.resolve("push-state.properties"); + Files.writeString(stateFile, "world/region/r.0.0.mca=not-a-valid-encoded-state\n"); + + CopyState state = CopyState.load(stateFile); + + assertEquals(0, state.size()); + } + + @Test + void savingLeavesNoTemporaryFileBehind() throws IOException { + Path stateFile = tmp.resolve("push-state.properties"); + CopyState state = CopyState.empty(); + state.put("world/region/r.0.0.mca", new RegionFileState(1, 2, "checksum")); + + state.save(stateFile); + + List leftovers; + try (var walk = Files.list(tmp)) { + leftovers = walk.filter(p -> p.getFileName().toString().endsWith(".tmp")).toList(); + } + assertTrue(leftovers.isEmpty()); + } + + @Test + void savingTwiceReplacesThePreviousContentAtomically() throws IOException { + Path stateFile = tmp.resolve("push-state.properties"); + CopyState first = CopyState.empty(); + first.put("world/region/r.0.0.mca", new RegionFileState(1, 2, "first")); + first.save(stateFile); + + CopyState second = CopyState.empty(); + second.put("world/region/r.0.0.mca", new RegionFileState(1, 2, "second")); + second.save(stateFile); + + CopyState reloaded = CopyState.load(stateFile); + assertEquals(1, reloaded.size()); + assertEquals("second", reloaded.get("world/region/r.0.0.mca").checksum()); + } +} diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/DimensionLayoutTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/DimensionLayoutTest.java new file mode 100644 index 0000000..d40bfeb --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/DimensionLayoutTest.java @@ -0,0 +1,64 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class DimensionLayoutTest { + + @TempDir + Path tmp; + + @Test + void onlyTheOverworldExistsByDefault() throws IOException { + Files.createDirectories(tmp.resolve("world/region")); + + List dims = DimensionLayout.forWorld(tmp, "world"); + + assertEquals(1, dims.size()); + assertEquals("world/region", dims.get(0).relativePrefix()); + assertEquals(tmp.resolve("world/region"), dims.get(0).sourceDir()); + } + + @Test + void netherAndEndAreIncludedWhenPresent() throws IOException { + Files.createDirectories(tmp.resolve("world/region")); + Files.createDirectories(tmp.resolve("world_nether/DIM-1/region")); + Files.createDirectories(tmp.resolve("world_the_end/DIM1/region")); + + List dims = DimensionLayout.forWorld(tmp, "world"); + + assertEquals(3, dims.size()); + assertTrue(dims.stream().anyMatch(d -> d.relativePrefix().equals("world/region"))); + assertTrue(dims.stream().anyMatch(d -> d.relativePrefix().equals("world_nether/DIM-1/region"))); + assertTrue(dims.stream().anyMatch(d -> d.relativePrefix().equals("world_the_end/DIM1/region"))); + } + + @Test + void aWorldThatDoesNotExistAtAllYieldsNoDirectories() { + assertEquals(List.of(), DimensionLayout.forWorld(tmp, "nonexistent")); + } +} diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/HttpPushNotifierTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/HttpPushNotifierTest.java new file mode 100644 index 0000000..d113437 --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/HttpPushNotifierTest.java @@ -0,0 +1,106 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import com.sun.net.httpserver.HttpServer; +import java.io.IOException; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; + +/** + * Proves {@link HttpPushNotifier} sends exactly the wire shape {@code PushController}/{@code + * PushReportRequest} (module {@code api}, package {@code net.onelitefeather.apus.api.rest.push}) + * actually expects: the token as a URL path segment (never a header, despite what an earlier + * version of {@code config.yml}'s comment claimed), and a JSON body with exactly {@code + * sourceName}/{@code version} -- the two fields {@code PushReportRequest} deserializes. Before + * this test (and the fix it locks in) existed, this class sent {@code tenant}/{@code + * worldName}/{@code fileCount}/{@code bytesUploaded} instead, which {@code PushController} would + * have rejected with a 400 (missing {@code sourceName}/{@code version}) on every real push -- + * exactly the kind of plugin/endpoint drift that only surfaces in production without a test like + * this one. + */ +class HttpPushNotifierTest { + + private HttpServer server; + private volatile String capturedPath; + private volatile String capturedBody; + + @AfterEach + void tearDown() { + if (server != null) { + server.stop(0); + } + } + + @Test + void sendsTheTokenAsAPathSegmentAndSourceNameAndVersionAsTheJsonBody() throws IOException { + startServer(204); + HttpPushNotifier notifier = new HttpPushNotifier(baseUrl(), "sh4r3d-t0ken"); + + notifier.notifyPushComplete(new PushSummary("survival-source", "2026-08-09T12-00-00Z", "world", 3, 42)); + + assertEquals("/api/push/sh4r3d-t0ken", capturedPath, "token must be a path segment, never a header"); + assertEquals("{\"sourceName\":\"survival-source\",\"version\":\"2026-08-09T12-00-00Z\"}", capturedBody); + } + + @Test + void aNonTwoXxResponseThrows() throws IOException { + startServer(400); + HttpPushNotifier notifier = new HttpPushNotifier(baseUrl(), "token"); + + assertThrows( + HttpPushNotifier.PushNotificationException.class, + () -> notifier.notifyPushComplete(new PushSummary("source", "v1", "world", 1, 1))); + } + + @Test + void anUnreachableApiThrowsWithoutLeakingTheTokenInTheMessage() { + HttpPushNotifier notifier = new HttpPushNotifier(URI.create("http://127.0.0.1:1"), "super-secret-token"); + + HttpPushNotifier.PushNotificationException e = assertThrows( + HttpPushNotifier.PushNotificationException.class, + () -> notifier.notifyPushComplete(new PushSummary("source", "v1", "world", 1, 1))); + assertTrue( + !e.getMessage().contains("super-secret-token"), + "the failure message must never echo the push token"); + } + + private void startServer(int statusCode) throws IOException { + server = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0); + server.createContext("/api/push/", exchange -> { + capturedPath = exchange.getRequestURI().getPath(); + capturedBody = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8); + byte[] response = new byte[0]; + exchange.sendResponseHeaders(statusCode, response.length == 0 ? -1 : response.length); + exchange.close(); + }); + server.start(); + } + + private URI baseUrl() { + return URI.create("http://" + server.getAddress().getHostString() + ":" + server.getAddress().getPort()); + } +} diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/IncrementalWorldCopierTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/IncrementalWorldCopierTest.java new file mode 100644 index 0000000..d212490 --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/IncrementalWorldCopierTest.java @@ -0,0 +1,189 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.FileTime; +import java.util.List; +import java.util.stream.Stream; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Covers the fachlicher Kern of this module: which region files count as "changed" between two + * runs, that {@link CopyState} correctly carries forward across separate {@link + * IncrementalWorldCopier} calls (simulating separate push cycles), and that a failure partway + * through a copy never leaves a half-written file in the staging directory. + */ +class IncrementalWorldCopierTest { + + private final IncrementalWorldCopier copier = new IncrementalWorldCopier(); + + @TempDir + Path tmp; + + private Path regionDir; + private Path stagingRoot; + + @BeforeEach + void setUp() throws IOException { + regionDir = Files.createDirectories(tmp.resolve("world/region")); + stagingRoot = tmp.resolve("staging"); + } + + @Test + void copiesEveryRegionFileOnTheFirstRun() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + writeRegionFile("r.0.1.mca", "beta"); + + CopyResult result = copier.copyChanged(dimensions(), stagingRoot, CopyState.empty()); + + assertEquals(2, result.copiedRelativePaths().size()); + assertEquals(0, result.unchangedCount()); + assertTrue(result.copiedRelativePaths().contains("world/region/r.0.0.mca")); + assertTrue(result.copiedRelativePaths().contains("world/region/r.0.1.mca")); + assertEquals("alpha", Files.readString(stagingRoot.resolve("world/region/r.0.0.mca"))); + assertEquals("beta", Files.readString(stagingRoot.resolve("world/region/r.0.1.mca"))); + } + + @Test + void secondRunWithNoChangesCopiesNothing() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + CopyState state = CopyState.empty(); + copier.copyChanged(dimensions(), stagingRoot, state); + + CopyResult second = copier.copyChanged(dimensions(), stagingRoot, state); + + assertTrue(second.isEmpty()); + assertEquals(1, second.unchangedCount()); + } + + @Test + void onlyTheChangedFileIsCopiedOnASecondRun() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + writeRegionFile("r.0.1.mca", "beta"); + CopyState state = CopyState.empty(); + copier.copyChanged(dimensions(), stagingRoot, state); + + // Modify only one file, moving both its content and its mtime forward. + writeRegionFile("r.0.1.mca", "beta-v2"); + bumpMtime(regionDir.resolve("r.0.1.mca")); + + CopyResult second = copier.copyChanged(dimensions(), stagingRoot, state); + + assertEquals(List.of("world/region/r.0.1.mca"), second.copiedRelativePaths()); + assertEquals(1, second.unchangedCount()); + assertEquals("beta-v2", Files.readString(stagingRoot.resolve("world/region/r.0.1.mca"))); + } + + @Test + void aTouchedFileWithUnchangedContentIsNotRecopied() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + CopyState state = CopyState.empty(); + copier.copyChanged(dimensions(), stagingRoot, state); + + // Rewrite with byte-identical content but a new mtime -- e.g. an atomic region-file + // rewrite that happens to produce the same bytes. + writeRegionFile("r.0.0.mca", "alpha"); + bumpMtime(regionDir.resolve("r.0.0.mca")); + + CopyResult second = copier.copyChanged(dimensions(), stagingRoot, state); + + assertTrue(second.isEmpty(), "identical content must not be re-copied even if mtime moved"); + // But the cheap-check fields must have been refreshed, so a genuinely-unrelated later + // touch doesn't get misdiagnosed against a stale mtime. + RegionFileState refreshed = state.get("world/region/r.0.0.mca"); + assertEquals(Files.getLastModifiedTime(regionDir.resolve("r.0.0.mca")).toMillis(), refreshed.lastModifiedMillis()); + } + + @Test + void stateCarriesForwardAcrossASimulatedProcessRestart() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + Path stateFile = tmp.resolve("push-state.properties"); + + CopyState first = CopyState.load(stateFile); + copier.copyChanged(dimensions(), stagingRoot, first); + first.save(stateFile); + + // Simulate a fresh process: reload state from disk instead of reusing the in-memory object. + CopyState reloaded = CopyState.load(stateFile); + CopyResult second = copier.copyChanged(dimensions(), stagingRoot, reloaded); + + assertTrue(second.isEmpty(), "reloaded state must still recognise the file as unchanged"); + } + + @Test + void nonMcaFilesInTheRegionDirectoryAreIgnored() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + Files.writeString(regionDir.resolve("r.0.0.mca.tmp"), "leftover"); + Files.writeString(regionDir.resolve("README.txt"), "not a region file"); + + CopyResult result = copier.copyChanged(dimensions(), stagingRoot, CopyState.empty()); + + assertEquals(List.of("world/region/r.0.0.mca"), result.copiedRelativePaths()); + } + + @Test + void anAbortedCopyLeavesNoTemporaryFileBehind() throws IOException { + writeRegionFile("r.0.0.mca", "alpha"); + // Make the staging destination directory impossible to create by occupying its path + // with a regular file -- Files.createDirectories(...) will then fail for every file + // under it, simulating an I/O failure partway through a batch. + Files.createDirectories(stagingRoot); + Files.writeString(stagingRoot.resolve("world"), "blocking file, not a directory"); + + assertTrue( + org.junit.jupiter.api.Assertions.assertThrows( + IOException.class, () -> copier.copyChanged(dimensions(), stagingRoot, CopyState.empty())) + .getMessage() + != null); + + // No stray *.tmp file anywhere under staging -- nothing "half" was left behind. + try (Stream walk = Files.exists(stagingRoot) ? Files.walk(stagingRoot) : Stream.empty()) { + assertFalse(walk.anyMatch(p -> p.getFileName().toString().endsWith(".tmp"))); + } + } + + private List dimensions() { + return List.of(new DimensionRegionDir("world/region", regionDir)); + } + + private void writeRegionFile(String name, String content) throws IOException { + Files.writeString( + regionDir.resolve(name), content, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING); + } + + /** + * Explicitly advances a file's mtime by 5 seconds instead of sleeping the test thread: some + * filesystems (notably ext4 with a 1s-granularity mount, or overlay filesystems used in CI + * containers) can otherwise report the same mtime for two writes issued within the same + * second, making a real sleep both slow and still not fully reliable. + */ + private static void bumpMtime(Path file) throws IOException { + FileTime current = Files.getLastModifiedTime(file); + Files.setLastModifiedTime(file, FileTime.fromMillis(current.toMillis() + 5000)); + } +} diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/PushCycleRunnerTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/PushCycleRunnerTest.java new file mode 100644 index 0000000..d2b5f66 --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/PushCycleRunnerTest.java @@ -0,0 +1,227 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * Exercises {@link PushCycleRunner}'s orchestration with fakes standing in for the parts that + * need a live Paper server or network access ({@link SaveCoordinator}, {@link WorldUploader}, + * {@link PushNotifier}) -- this is the "what's testable without a running server" half of the + * Bukkit-facing code the phase 6 task report describes; {@link BukkitSaveCoordinator} itself is + * not exercised here. + */ +class PushCycleRunnerTest { + + @TempDir + Path tmp; + + private Path regionDir; + private Path stagingRoot; + private Path stateFile; + private WorldPushConfig config; + private FakeSaveCoordinator saveCoordinator; + private FakeUploader uploader; + private FakeNotifier notifier; + + @BeforeEach + void setUp() throws IOException { + Path serverRoot = tmp.resolve("server"); + regionDir = Files.createDirectories(serverRoot.resolve("world/region")); + stagingRoot = tmp.resolve("staging"); + stateFile = tmp.resolve("push-state.properties"); + config = WorldPushConfig.from(configSource()); + saveCoordinator = new FakeSaveCoordinator(); + uploader = new FakeUploader(); + notifier = new FakeNotifier(); + this.serverRoot = serverRoot; + } + + private Path serverRoot; + + @Test + void happyPathSavesUploadsNotifiesAndPersistsState() throws IOException { + Files.writeString(regionDir.resolve("r.0.0.mca"), "alpha"); + + newRunner().runCycle(); + + assertEquals(List.of("disableAutoSave", "forceSave", "enableAutoSave"), saveCoordinator.calls); + assertEquals(1, uploader.uploaded.size()); + assertEquals("staging/world/region/r.0.0.mca", uploader.uploaded.get(0).s3Key()); + assertEquals(1, notifier.summaries.size()); + assertEquals( + new PushSummary("survival-source", "2026-08-09T12-00-00Z", "world", 1, 5), + notifier.summaries.get(0)); + assertTrue(Files.isRegularFile(stateFile), "state must be persisted after a successful cycle"); + } + + @Test + void autoSaveIsReEnabledEvenIfForceSaveFails() throws IOException { + Files.writeString(regionDir.resolve("r.0.0.mca"), "alpha"); + saveCoordinator.failForceSave = true; + + assertThrows(RuntimeException.class, () -> newRunner().runCycle()); + + assertEquals(List.of("disableAutoSave", "forceSave", "enableAutoSave"), saveCoordinator.calls); + assertTrue(uploader.uploaded.isEmpty(), "must not upload if the save step itself failed"); + } + + @Test + void noChangesMeansNoUploadAndNoNotificationButStateIsStillSaved() throws IOException { + Files.writeString(regionDir.resolve("r.0.0.mca"), "alpha"); + newRunner().runCycle(); + uploader.uploaded.clear(); + notifier.summaries.clear(); + + newRunner().runCycle(); + + assertTrue(uploader.uploaded.isEmpty()); + assertTrue(notifier.summaries.isEmpty()); + } + + @Test + void aFailedUploadLeavesThePersistedStateUnchangedForRetry() throws IOException { + Files.writeString(regionDir.resolve("r.0.0.mca"), "alpha"); + uploader.failUploads = true; + + assertThrows(RuntimeException.class, () -> newRunner().runCycle()); + + assertFalse(Files.isRegularFile(stateFile), "a failed cycle must not persist state"); + assertTrue(notifier.summaries.isEmpty(), "must not notify if the upload failed"); + + // A retried cycle (fresh runner, uploads succeeding this time) must still see the file + // as changed and upload it -- nothing was silently marked done by the failed attempt. + uploader.failUploads = false; + newRunner().runCycle(); + assertEquals(1, uploader.uploaded.size()); + assertEquals(1, notifier.summaries.size()); + } + + @Test + void aFailedNotificationLeavesThePersistedStateUnchangedForRetry() throws IOException { + Files.writeString(regionDir.resolve("r.0.0.mca"), "alpha"); + notifier.failNotifications = true; + + assertThrows(RuntimeException.class, () -> newRunner().runCycle()); + + assertFalse(Files.isRegularFile(stateFile), "a failed cycle must not persist state"); + // The upload itself did happen -- only the state advancement is what got rolled back. + assertEquals(1, uploader.uploaded.size()); + } + + private static final Clock FIXED_CLOCK = + Clock.fixed(Instant.parse("2026-08-09T12:00:00Z"), ZoneOffset.UTC); + + private PushCycleRunner newRunner() { + return new PushCycleRunner( + new IncrementalWorldCopier(), saveCoordinator, uploader, notifier, serverRoot, stagingRoot, stateFile, + config, FIXED_CLOCK); + } + + private static ConfigSource configSource() { + Map values = new HashMap<>(); + values.put("world-name", "world"); + values.put("tenant", "acme"); + values.put("world-source-name", "survival-source"); + values.put("push-token", "secret-token"); + values.put("s3.endpoint", "https://s3.example.org"); + values.put("s3.bucket", "apus-worlds"); + values.put("s3.access-key", "access-key"); + values.put("s3.secret-key", "secret-key"); + values.put("apus.api-base-url", "https://apus.example.org"); + return new ConfigSource() { + @Override + public String getString(String path) { + return values.get(path); + } + + @Override + public long getLong(String path, long defaultValue) { + return defaultValue; + } + }; + } + + private static final class FakeSaveCoordinator implements SaveCoordinator { + final List calls = new ArrayList<>(); + boolean failForceSave; + + @Override + public void disableAutoSave() { + calls.add("disableAutoSave"); + } + + @Override + public void forceSave() { + calls.add("forceSave"); + if (failForceSave) { + throw new RuntimeException("simulated save failure"); + } + } + + @Override + public void enableAutoSave() { + calls.add("enableAutoSave"); + } + } + + private record Upload(Path localFile, String s3Key) {} + + private static final class FakeUploader implements WorldUploader { + final List uploaded = new ArrayList<>(); + boolean failUploads; + + @Override + public void upload(Path localFile, String s3Key) { + if (failUploads) { + throw new RuntimeException("simulated upload failure"); + } + uploaded.add(new Upload(localFile, s3Key)); + } + } + + private static final class FakeNotifier implements PushNotifier { + final List summaries = new ArrayList<>(); + boolean failNotifications; + + @Override + public void notifyPushComplete(PushSummary summary) { + if (failNotifications) { + throw new RuntimeException("simulated notification failure"); + } + summaries.add(summary); + } + } +} diff --git a/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/WorldPushConfigTest.java b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/WorldPushConfigTest.java new file mode 100644 index 0000000..c530e5d --- /dev/null +++ b/paper-worldpush/src/test/java/net/onelitefeather/apus/paper/WorldPushConfigTest.java @@ -0,0 +1,173 @@ +/** + * Apus - render and host BlueMap maps on Kubernetes. + * Copyright (C) 2026 OneLiteFeather and contributors + *

+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + *

+ * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + *

+ * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ +package net.onelitefeather.apus.paper; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +import java.util.HashMap; +import java.util.Map; +import org.junit.jupiter.api.Test; + +class WorldPushConfigTest { + + @Test + void validConfigParsesEveryField() { + WorldPushConfig config = WorldPushConfig.from(source(fullConfig())); + + assertEquals("world", config.worldName()); + assertEquals("acme", config.tenant()); + assertEquals("survival-source", config.sourceName()); + assertEquals("secret-token", config.pushToken()); + assertEquals("apus-worldpush-staging", config.stagingDirectory()); + assertEquals("https://s3.example.org", config.s3Endpoint()); + assertEquals("apus-worlds", config.s3Bucket()); + assertEquals("us-east-1", config.s3Region()); + assertEquals("access-key", config.s3AccessKey()); + assertEquals("secret-key", config.s3SecretKey()); + assertEquals("staging/", config.s3StagingPrefix()); + assertEquals("https://apus.example.org", config.apusApiBaseUrl().toString()); + assertEquals(30, config.intervalMinutes()); + } + + @Test + void stagingPrefixIsNormalisedToEndWithASlash() { + Map values = fullConfig(); + values.put("s3.staging-prefix", "staging/acme"); + + WorldPushConfig config = WorldPushConfig.from(source(values)); + + assertEquals("staging/acme/", config.s3StagingPrefix()); + } + + @Test + void missingWorldNameFailsFast() { + Map values = fullConfig(); + values.remove("world-name"); + + WorldPushConfig.ConfigurationException e = + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + org.junit.jupiter.api.Assertions.assertTrue(e.getMessage().contains("world-name")); + } + + @Test + void missingPushTokenFailsFast() { + Map values = fullConfig(); + values.remove("push-token"); + + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + } + + @Test + void missingSourceNameFailsFast() { + Map values = fullConfig(); + values.remove("world-source-name"); + + WorldPushConfig.ConfigurationException e = + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + org.junit.jupiter.api.Assertions.assertTrue(e.getMessage().contains("world-source-name")); + } + + @Test + void blankS3CredentialsFailFast() { + Map values = fullConfig(); + values.put("s3.access-key", " "); + + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + } + + @Test + void malformedApiBaseUrlFailsFast() { + Map values = fullConfig(); + values.put("apus.api-base-url", "not a url"); + + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + } + + @Test + void relativeApiBaseUrlFailsFast() { + Map values = fullConfig(); + values.put("apus.api-base-url", "/api"); + + assertThrows(WorldPushConfig.ConfigurationException.class, () -> WorldPushConfig.from(source(values))); + } + + @Test + void zeroOrNegativeIntervalFailsFast() { + WorldPushConfig.ConfigurationException e = assertThrows( + WorldPushConfig.ConfigurationException.class, + () -> WorldPushConfig.from(new ConfigSource() { + @Override + public String getString(String path) { + return fullConfig().get(path); + } + + @Override + public long getLong(String path, long defaultValue) { + return "schedule.interval-minutes".equals(path) ? 0 : defaultValue; + } + })); + org.junit.jupiter.api.Assertions.assertTrue(e.getMessage().contains("interval-minutes")); + } + + @Test + void defaultsApplyWhenOptionalKeysAreAbsent() { + Map values = fullConfig(); + values.remove("staging-directory"); + values.remove("s3.region"); + values.remove("s3.staging-prefix"); + + WorldPushConfig config = WorldPushConfig.from(source(values)); + + assertEquals("apus-worldpush-staging", config.stagingDirectory()); + assertEquals("us-east-1", config.s3Region()); + assertEquals("staging/", config.s3StagingPrefix()); + assertEquals(30, config.intervalMinutes()); + } + + private static Map fullConfig() { + Map values = new HashMap<>(); + values.put("world-name", "world"); + values.put("tenant", "acme"); + values.put("world-source-name", "survival-source"); + values.put("push-token", "secret-token"); + values.put("staging-directory", "apus-worldpush-staging"); + values.put("s3.endpoint", "https://s3.example.org"); + values.put("s3.bucket", "apus-worlds"); + values.put("s3.region", "us-east-1"); + values.put("s3.access-key", "access-key"); + values.put("s3.secret-key", "secret-key"); + values.put("s3.staging-prefix", "staging/"); + values.put("apus.api-base-url", "https://apus.example.org"); + return values; + } + + private static ConfigSource source(Map values) { + return new ConfigSource() { + @Override + public String getString(String path) { + return values.get(path); + } + + @Override + public long getLong(String path, long defaultValue) { + return "schedule.interval-minutes".equals(path) ? 30 : defaultValue; + } + }; + } +} diff --git a/runner/src/test/java/net/onelitefeather/apus/runner/IngestRenderContractTest.java b/runner/src/test/java/net/onelitefeather/apus/runner/IngestRenderContractTest.java index 5012b88..e1c6772 100644 --- a/runner/src/test/java/net/onelitefeather/apus/runner/IngestRenderContractTest.java +++ b/runner/src/test/java/net/onelitefeather/apus/runner/IngestRenderContractTest.java @@ -71,9 +71,15 @@ class IngestRenderContractTest { private static final String SOURCE_KEY = "v1.zip"; private static final String BUNDLE_TENANT = "acme"; + // The owning WorldSource's name -- required by IngestConfig.ENV_BUNDLE_SOURCE_NAME and, per + // BundlePath, the second path segment (tenant/sourceName/worldId/version). This constant and + // the env var below were missing from this test even after that requirement was introduced; + // fixed as a drive-by while touching IngestConfig for phase 6 (unrelated to push/upload). + private static final String BUNDLE_SOURCE_NAME = "demo-source"; private static final String BUNDLE_WORLD_ID = "spawn"; private static final String BUNDLE_VERSION = "v1"; - private static final String BUNDLE_PATH = BUNDLE_TENANT + "/" + BUNDLE_WORLD_ID + "/" + BUNDLE_VERSION; + private static final String BUNDLE_PATH = + BUNDLE_TENANT + "/" + BUNDLE_SOURCE_NAME + "/" + BUNDLE_WORLD_ID + "/" + BUNDLE_VERSION; // What LayoutDetector.detect must normalise a Bukkit-layout source's sibling folders // (world, world_nether, world_the_end) to -- the "core of normalisation" the phase 2b plan @@ -183,6 +189,7 @@ private static int runIngest(MinIOContainer minio, Path workDir) { env.put(IngestConfig.ENV_SOURCE_VERSION, SOURCE_KEY); env.put(IngestConfig.ENV_BUNDLE_BUCKET, MinioFixtures.WORLD_BUCKET); env.put(IngestConfig.ENV_BUNDLE_TENANT, BUNDLE_TENANT); + env.put(IngestConfig.ENV_BUNDLE_SOURCE_NAME, BUNDLE_SOURCE_NAME); env.put(IngestConfig.ENV_BUNDLE_WORLD_ID, BUNDLE_WORLD_ID); env.put(IngestConfig.ENV_BUNDLE_VERSION, BUNDLE_VERSION); env.put(IngestConfig.ENV_S3_ENDPOINT, minio.getS3URL()); @@ -259,19 +266,35 @@ private static void assertManifestIsComplete(BundleManifest manifest) { } /** - * Independently cross-checks the manifest's claims against what MinIO actually holds: exactly - * the expected six region objects plus the manifest itself, no more, no less, and -- read + * Content {@link net.onelitefeather.apus.ingest.BundleWriter} writes alongside the mandatory + * region files, exactly as documented in the design spec's bundle layout (see {@code + * docs/superpowers/specs/2026-08-08-apus-design.md}, "worlds/<tenant>/<world-id>/ + * <version>/" section): the world's {@code level.dat} at the bundle root, and, per + * dimension and only "falls vorhanden" (if present in the source), {@code entities/}/{@code + * poi/} region-shaped files. This fixture's source world never has entities/poi siblings, so + * only {@code level.dat} shows up in practice today, but the pattern is written to already + * cover entities/poi too, so a future fixture that exercises them does not have to touch this + * assertion again. + */ + private static final Pattern DOCUMENTED_SIDECAR_KEY = + Pattern.compile("level\\.dat|dimensions/[^/]+/(entities|poi)/r\\.-?\\d+\\.-?\\d+\\.mca"); + + /** + * Independently cross-checks the manifest's claims against what MinIO actually holds: every + * mandatory object (the manifest itself, plus every region file the manifest lists) must be + * present, and anything beyond that must be bundle content the design spec documents ({@link + * #DOCUMENTED_SIDECAR_KEY}) -- never a stray or misplaced object. Also checked -- read * straight from real object timestamps, not from a fake client's call log the way {@code * BundleWriterTest} already proves this in isolation -- the manifest is the object with the * latest {@code lastModified} of the bundle, i.e. it really was written last against a real * S3-compatible store, not merely in a unit test double. */ private static void assertRealBucketListingMatchesTheManifestWithManifestWrittenLast(Network network) { - Set expectedKeys = new LinkedHashSet<>(); - expectedKeys.add("manifest.json"); + Set requiredKeys = new LinkedHashSet<>(); + requiredKeys.add("manifest.json"); for (String dimension : LOGICAL_DIMENSIONS) { for (String regionFile : REGION_FILE_NAMES) { - expectedKeys.add("dimensions/" + dimension + "/region/" + regionFile); + requiredKeys.add("dimensions/" + dimension + "/region/" + regionFile); } } @@ -301,7 +324,25 @@ private static void assertRealBucketListingMatchesTheManifestWithManifestWritten lastModifiedByKey.put(keyMatcher.group(1), Instant.parse(lastModifiedMatcher.group(1))); } - assertEquals(expectedKeys, lastModifiedByKey.keySet(), "bucket must hold exactly the bundle's own objects:\n" + logs); + Set actualKeys = lastModifiedByKey.keySet(); + Set missingKeys = new LinkedHashSet<>(requiredKeys); + missingKeys.removeAll(actualKeys); + assertTrue( + missingKeys.isEmpty(), + "bucket must hold every mandatory bundle object (manifest.json plus every region file the " + + "manifest lists); missing: " + missingKeys + "\n" + logs); + + Set unexpectedKeys = new LinkedHashSet<>(); + for (String key : actualKeys) { + if (!requiredKeys.contains(key) && !DOCUMENTED_SIDECAR_KEY.matcher(key).matches()) { + unexpectedKeys.add(key); + } + } + assertTrue( + unexpectedKeys.isEmpty(), + "bucket must hold only the bundle's own objects -- mandatory region files/manifest.json plus " + + "sidecar content the design spec documents (level.dat, entities/, poi/); unexpected: " + + unexpectedKeys + "\n" + logs); Instant manifestWrittenAt = lastModifiedByKey.get("manifest.json"); for (Map.Entry entry : lastModifiedByKey.entrySet()) { diff --git a/settings.gradle.kts b/settings.gradle.kts index 85d086d..e39482f 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,11 +1,14 @@ rootProject.name = "Apus" -include("telemetry-addon", "runner", "operator", "ingest", "api") +include("telemetry-addon", "runner", "operator", "ingest", "api", "paper-worldpush") dependencyResolutionManagement { repositories { mavenCentral() maven("https://repo.bluecolored.de/releases") + // paper-api, for :paper-worldpush -- see that module's own note in this file for why + // it depends on a foreign version track instead of the rest of this catalog. + maven("https://repo.papermc.io/repository/maven-public/") } versionCatalogs { create("libs") { @@ -133,6 +136,11 @@ dependencyResolutionManagement { library("fabric8.server.mock", "io.fabric8", "kubernetes-server-mock").versionRef("fabric8") library("aws.sdk.bom", "software.amazon.awssdk", "bom").versionRef("aws-sdk") + // software.amazon.awssdk.services.s3.presigner.S3Presigner (used by the `api` module's + // POST /api/uploads, design spec §11.1, to hand out presigned multipart-upload part + // URLs) ships inside this same artifact in this SDK major version -- verified directly + // against the resolved s3-2.46.7.jar on 2026-08-09; there is no separate + // `s3-presigner` artifact to depend on (an earlier SDK version did have one). library("aws.sdk.s3", "software.amazon.awssdk", "s3").withoutVersion() library("jackson.bom", "com.fasterxml.jackson", "jackson-bom").versionRef("jackson") @@ -140,6 +148,22 @@ dependencyResolutionManagement { library("cron.utils", "com.cronutils", "cron-utils").versionRef("cron-utils") + // Paper API, for :paper-worldpush (phase 6, task 1) -- the plugin that lets a live + // Paper server push its own world instead of Apus pulling it. Deliberately its own + // version() entry rather than reusing anything above: like bluemap-core/bluemap-api, + // this tracks a fast-moving third-party project (see §4 of the design spec, "eigene + // Release-Spur"), not this repo's own version. Pinned to a specific stable build + // rather than the floating "26.2.build.+" range PaperMC's own setup docs show, to + // keep this build reproducible -- the same reasoning already applied to every other + // pinned version in this catalog. Verified against + // https://repo.papermc.io/repository/maven-public/io/papermc/paper/paper-api/maven-metadata.xml + // on 2026-08-09: 26.2.build.111-stable is the newest build on the "stable" channel + // (id 111, 2026-08-07). Minecraft/Paper 26.2 is the current version line (PaperMC + // moved off the old 1.21.x scheme); api-version in paper-plugin.yml uses the short + // "26.2" form the same metadata/docs use. + version("paper-api", "26.2.build.111-stable") + library("paper.api", "io.papermc.paper", "paper-api").versionRef("paper-api") + plugin("spotless", "com.diffplug.spotless").versionRef("spotless") plugin("shadow", "com.gradleup.shadow").versionRef("shadow") }