diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index cc78768..6369644 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -79,3 +79,29 @@ jobs: run: | python3 tools/test_check_version_floors.py python3 tools/check-version-floors.py + + # Includes a dry run of the next release against the real CHANGELOG.md and + # changelog.d/, so a fragment or marker that would break it fails here first. + - name: Changelog collect (stdlib only) + run: python3 tools/test_changelog_collect.py + + # Every PR that added its entry under the shared `## [Unreleased]` heading + # conflicted with every other open PR that did the same. Entries go in + # changelog.d/ (one file each); only a release/* branch rewrites CHANGELOG.md. + changelog-fragments: + name: CHANGELOG.md is edited only by releases + if: github.event_name == 'pull_request' && !startsWith(github.head_ref, 'release/') + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + with: + fetch-depth: 2 + persist-credentials: false + + - name: Changelog entries are fragments in changelog.d/ + run: | + # HEAD is GitHub's test merge of the PR into its base, so HEAD^1 is the base tip. + if ! git diff --quiet HEAD^1 HEAD -- CHANGELOG.md; then + echo "::error file=CHANGELOG.md::Put the entry in a new file under changelog.d/ instead (see changelog.d/README.md). CHANGELOG.md changes only on a release/* branch." + exit 1 + fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a4a1d9..4d8a2d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,516 +4,10 @@ All notable changes to the CacheKit Protocol Specification. ## [Unreleased] -### Interop mode — `ns` and `nsapi` are reserved namespaces (LAB-5876) +New entries go in [`changelog.d/`](changelog.d/README.md), one file per change, and are +collected into a version section here at each release. -- [`spec/interop-mode.md` → Segment grammar](spec/interop-mode.md#segment-grammar): - `namespace` MUST NOT be `ns` or `nsapi`. The segment pattern admitted both, but the - resulting key starts `ns:` / `nsapi:`, which the server parses as namespace-prefixed - ([cache-key-format.md → Server-Side Requirements](spec/cache-key-format.md#server-side-requirements)): - rejected when the operation contains `.`, otherwise scoped to a namespace named after - the operation. The reservation is exact-match and namespace-only; `ns` and `nsapi` - stay valid operations. SDKs reject a reserved namespace at decoration / registration - time, on every backend. **Breaking for any deployment that uses namespace `ns` or - `nsapi`, on any backend:** it now raises at startup; migrate by renaming the namespace - (a full cache miss for that namespace). -- [`test-vectors/interop-mode.json`](test-vectors/interop-mode.json) 1.1.0: two error - vectors (`reject_reserved_namespace_ns`, `reject_reserved_namespace_nsapi`) and one key - vector (`reservation_scope`: namespace `nsapix`, operation `nsapi`) that pins the - reservation as namespace-only and exact-match. Counts: 34 key, 11 error. -- SaaS Considerations no longer calls the grammar a strict subset of what the server - accepts: the grammar admits `..` inside a segment, which the server rejects. -- SDK feature matrix: the "Test vectors in CI" cells note that fixture 1.1.0 is not yet - in any released SDK, linking the SDK PRs that vendor it. - `tools/interop-reference.py` builds every key vector through its validating - `interop_key`; `tools/interop-crosscheck.mjs` checks the segment grammar on key vectors - as well as error vectors, with the reserved names hard-coded rather than read from the - fixture. - -### Wire format — vendored-fixture coverage note corrected (LAB-1750) - -- [`spec/wire-format.md`](spec/wire-format.md) no longer says `cachekit-core` vendors - fixture 1.1.0. It pins 1.1.1, so `width_boundary_bin16_bin` has a canonical-writer - (`lz4_flex`) compressed-byte and xxh3-64 checksum check. The section now states the rule - for anyone vendoring the fixture: derive each `*_bin` twin's expected marker from its - decoded `compressed_data` length, never assume bin8 or accept any `bin` width. - -### Encryption — default-tenant conformance vector (LAB-4666) - -- [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `default_tenant` - block (file version 1.2.0): tenant `"default"` under the main master key, one entry - (`default_tenant_interop`) sealed at the interop `single_int` key over the - `issue_example_object` plaintext, so it loads directly as an interop-mode entry. An SDK - that decrypts it through its `secure` preset with **no tenant configured** has - demonstrated [intent-presets.md § Master Key Input](spec/intent-presets.md#master-key-input) - rule 5 byte-for-byte. `tools/encryption-verify.py` checks the block (block-level derived-key fingerprint, AAD - with `"default"` as component 1, seal); append-only like the main set. -- Conformance table: Python's default-`tenant_id` row flips to ✅ (cachekit-py LAB-4666 — - implicit tenant is the literal `"default"`; the persisted per-host deployment UUID is - gone, explicit `deployment_uuid` / `CACHEKIT_DEPLOYMENT_UUID` remain as the override). - -### SaaS API - -- **`X-CacheKit-Fresh-For` remaining-freshness response header (LAB-557).** - `GET /v1/cache/{key}` `200 OK` responses carry the entry's remaining freshness - in whole seconds, so SDK local caches (L1) bound backfill to - `min(local_ttl, fresh_for)` instead of restarting the freshness clock at - time-of-read — which let an entry read near the end of its window be served - locally past `fresh_until` (and, with a stale-grace window, past `evict_at`). - Additive and backward compatible: an absent header means legacy behaviour on - both sides. Spec: [saas-api.md → Remaining Freshness](spec/saas-api.md#remaining-freshness). - - **Emission.** Sent on every `GET` `200 OK` for a bounded entry; `0` on - stale-window responses (and legal on `fresh` in the final sub-second); - omitted for no-expiry entries and by pre-signal servers — both mean "no - server-side bound", same SDK action. Never sent on `HEAD`, and a `HEAD` - response MUST NOT create or extend a local bound. - - **Re-serving tiers** decay the value and MUST NOT re-stamp it; they may omit - it only on positive knowledge of no expiry, and emit `0` otherwise; a - positive value only follows a `fresh` (or unlabelled) positive source or an - accepted write with a positive effective TTL. Coherence windows of composed - tiers add up, and the revocation-propagation bound is stated as that sum - plus the local bound, transit and clock error; `evict_at` is the store's - bound, not an end-to-end one. - - **SDK consumption.** The value is a hard local service bound — once it - elapses the local copy MUST NOT be served in any form, and `0` forbids - backfill. It MUST be 1–7 ASCII digits and at most `2,592,000`; anything - else is `0`. Local caches MUST NOT backfill a `stale`-labelled response at - all. Local deadlines SHOULD use a clock that counts across suspend. -- **`Cache-Control: no-store` and `Vary: Authorization` on every response.** - The cache key carries no tenant, so byte-identical URLs across tenants made - heuristic HTTP caching (RFC 9111 §4.2.2) a cross-tenant read. CacheKit-operated - caching tiers MUST partition by tenant. -- **Effective TTL.** A write's effective TTL is `X-CacheKit-TTL` when present, - otherwise the deprecated `X-TTL`; a write stores a no-expiry entry only when - it carries neither. The no-expiry rule, the stale-window requirement and the - tier rules all read the effective TTL, so a legacy `X-TTL: 60` write is no - longer mistaken for an immortal one. -- **No-expiry follow-ons** to the contract in the LAB-677 entry below. - `GET /v1/cache/{key}/ttl` returns `200 {"ttl": null}` for a no-expiry key - (mixed-reader caveat for SDKs that predate `null`), so its `404` now means - only "key absent"; an SDK TTL read MAY still collapse both to its null value. - A revalidation `PUT` MUST re-send the TTL as well as the stale window, or it - stores a no-expiry entry. Keys whose TTL is a revocation boundary MUST be - stored with an explicit TTL. The 30-day maximum bounds a stated TTL's value - range, not an entry's storage lifetime. - -### Intent presets — canonical preset contract (LAB-514) - -- New normative [`spec/intent-presets.md`](spec/intent-presets.md): what `minimal` / - `production` / `secure` / `io` MUST configure in every SDK. Decisions: finite default - TTLs (300 / 600 / 600 / 3 600 s — Python's cache-forever default is the outlier); L1 on - for every preset and ciphertext-only on `secure` (ratifies the 2025-11-13 cachekit-py - decision cross-SDK); no MUST on integrity checksums (the storage container is - SDK-internal, protocol#11); reliability stack default-on for `production`/`secure`/`io`; - `secure` is the canonical name in every SDK (Rust's `CacheKit::encrypted` is a tracked - non-conformance, LAB-4651); the encrypted preset MUST take a hex key and fall back to - `CACHEKIT_MASTER_KEY`; **`CACHEKIT_MASTER_KEY` is a key source, not an activation - switch** — it MUST NOT turn encryption on for `minimal`/`production`/`io`, and no - constructor is exempt (Python's fleet-wide auto-detect and Rust's `from_env()` - presence-activation are the outliers); an explicit encryption option MUST encrypt - every operation or be rejected; the default `tenant_id` is `"default"` and MUST be - identical for HKDF and AAD; no SDK MAY offer a process-wide default-TTL override; - `io` takes its API key by argument **or** `CACHEKIT_API_KEY`; explicit arguments - that a preset does not support MUST be rejected, never dropped. -- Per-SDK conformance table (code-verified 2026-09-22 against `main`: py `2f7c979`, - rs `6587ce9`, ts `379847c`) with one alignment ticket per ❌; TypeScript's only ❌ is - the HKDF-vs-AAD `tenant_id` mismatch. -- [`spec/encryption.md`](spec/encryption.md#master-key) Master Key table: minimum length - corrected from 16 bytes to **32 bytes (64 hex chars)** — every SDK enforces 32 at the - configuration boundary; 16 is the HKDF core's IKM floor and was never user-facing. -- [Feature matrix](sdk-feature-matrix.md#intent-preset-semantics-parity-not-presence) - intent-preset section now links the spec; README spec index gains the row. - -### Encryption — keyring conformance vectors + status reconciliation (LAB-687) - -- [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `keyring` - block: two master keys (`k1`, `k2`) for tenant `keyring-conformance`, one entry - sealed under each, and per-vector `key_fingerprint_hex` — the fingerprint of the - HKDF-derived per-tenant encryption key, as cachekit-py stores it. Frozen names - `encrypted_with_k1` / `encrypted_with_k2`; append-only like the main set. -- [`tools/encryption-verify.py`](tools/encryption-verify.py) enforces - [`spec/encryption.md` § Key Rotation (Keyring)](spec/encryption.md#key-rotation-keyring): - `[k2, k1]` decrypts both at the declared entry, `[k2]` alone rejects the k1 - entry, and the stored fingerprint selects the derived key — a master-key - fingerprint cannot select. Entry derivation, metadata, AAD and selection run - in the stdlib lane; only the decrypt attempts need `cryptography`. -- New [`tools/test_encryption_verify.py`](tools/test_encryption_verify.py) mutation - suite runs ahead of the verifier in both CI lanes (same doctrine as the - wire-format guard): every keyring guard is proven to go red by poisoning a copy - of the fixture. Added after the LAB-687 panel found three vacuous passes in the - first revision. -- Status banners reconciled with shipped code: `spec/encryption.md` and - `decisions/key-rotation.md` no longer say "not yet implemented"; the - [feature matrix](sdk-feature-matrix.md#encryption) Key rotation row is ✅ for - Python 0.18.0+ (cachekit-py#261) and 🚧 unreleased for Rust and TypeScript — - cachekit-rs#63 and cachekit-ts#103 are merged but absent from crates.io 0.7.0 - and npm 0.1.5 (artifacts inspected 2026-09-22). -- `decisions/key-rotation.md` compromise runbook: flush at cut-over **and** again - once the last old-key writer has stopped — a deferred single flush leaves the - whole compromised corpus readable for the rollout window; the decrypt-only - list must be explicitly empty, since an omitted list falls back to - `CACHEKIT_PREVIOUS_MASTER_KEYS`. - -### SaaS API — `401` is an authoritative verdict; auth backend faults are `503` (LAB-4093) - -- [`spec/saas-api.md`](spec/saas-api.md) Error Handling: `401` is emitted only - for an authoritative denial — a missing or malformed `Authorization` header, - or the key store saying the key is unknown, revoked, or its tenant suspended or soft-deleted. - A backend fault while resolving the key (auth cache / database) is a `503` - with `Retry-After`, so SDKs retry under the existing Transient class instead - of surfacing a transient blip as "invalid API key" (Permanent, never retried). - Documents the cache-worker behaviour shipped in - [cachekit-io/saas#380](https://github.com/cachekit-io/saas/pull/380); no - SDK change — `503` already classifies as Transient in all three. - -### SDK feature matrix — TypeScript `cache.secure.wrap()` now fails closed (LAB-513) - -- [`sdk-feature-matrix.md`](sdk-feature-matrix.md): the Encryption row "Does the - `secure` API enforce encryption?" flips ❌ → ✅ for TypeScript. Both - `cache.secure.wrap()` and `cache.withExecutionContext(ctx).secure.wrap()` now - throw `ConfigurationError` at wrap time on any instance without `encryption` - configured ([cachekit-ts#123](https://github.com/cachekit-io/cachekit-ts/pull/123)); - before, both were unconditional aliases for `wrap()`, so on an instance - without configured encryption a "secure" registration stored plaintext - (CWE-311) — on an encrypted instance they always encrypted. All three SDKs - now refuse a missing key on the secure entry point — py raises at decoration - time, rs `secure()` returns `Err`, ts throws at wrap time — with no opt-in to - run the secure entry point unencrypted in any of them; ts callers who want - plaintext call `wrap()` explicitly. The "Intent-preset - semantics" warning is rewritten to the enforced contract, the "cells that - reversed" summary and the Rust builder-stub cross-reference are updated to - match, and the stale `cache-core.ts:832` / `:873` / `:486`, `cache.ts:87` and - `intents-core.ts:240` references are replaced with current ones. - - -### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) - -- LZ4 compressed bytes are **not canonical** across conforming block encoders. - [`spec/wire-format.md`](spec/wire-format.md) now states this explicitly - (new "Compressed-byte reproducibility" section, mirroring interop v2's - doctrine): `compressed_data` conformance is read-side only, a **non-canonical** - writer is never judged non-conforming for differing from the pinned bytes - (byte-comparison as a declared-divergence tripwire remains allowed), and - only the canonical writer (`lz4_flex` via `cachekit-core`) has enforced - byte-reproducibility. The `large_compressible` / `large_compressible_bin` - pair is marked **known encode-divergent, decode-verified only** under the - spec's own reference liblz4 mapping — `lz4.block.compress(store_size=False)` - emits a 14 B block where the fixture pins `lz4_flex`'s 15 B. Found by - execution during the LAB-868 panel review; resolves the trust bug of a - fixture implying a reproducibility property the reference toolchain cannot - produce. Regeneration was rejected: every envelope-using SDK compresses - through `cachekit-core`'s `lz4_flex` (`cachekit-rs` writes plain MessagePack - with no envelope — spec 'SDK Storage Containers (auto mode)'), whose CI asserts re-encode byte-identity, so - re-pinning to liblz4 output would break the canonical writer and merely swap - which compressor diverges. -- [`tools/wire-format-reference.py`](tools/wire-format-reference.py) `verify` - gains an optional `lz4` leg (the dependency was already installed in CI's - optional-deps step): liblz4 MUST decompress every pinned `compressed_data` - to the pinned input; encoder agreement is asserted only as a drift - tripwire against `LZ4_ENCODE_DIVERGENT`, never as a per-vector conformance rule. The CI invocation now passes `--require-extras` - (precedent: `encryption-verify.py --require-seal`) so a dependency drift - cannot silently turn the deeper checks off. Fixture bytes untouched - (version stays 1.1.1) — no downstream SDK re-vendors required. -- Expert-panel hardening of the same verifier (crypto/protocol gate; every item - below was reproduced by poisoning the fixture and re-run after the fix): - - `original_size` is now checked against `len(input_hex)`, not just the - co-located `input_size` field. Both declared sizes live *in* the file under - test, so a regeneration bug that inflates them drifts them together and the - old check still passed — a vector declaring 100 MB for 16 bytes of input - verified green, and liblz4 did not catch it because - `decompress(uncompressed_size=…)` sizes the output buffer rather than - asserting the length. Runs on both CI legs (stdlib and optional-deps). - - **Both** commands refuse to run under `-O`/`PYTHONOPTIMIZE`: every conformance - check is an `assert`, so an optimised `verify` reported "all 7 vector pairs - verified" against a poisoned fixture, and an optimised `generate` rewrote the - fixture with its input checks stripped. The guard is at module scope, not in - `main()`, because a CLI-only guard is bypassed by importing the module and - calling `verify()` directly — which the regression harness's `importlib` - probe does, and which is how the sibling tools load each other's codecs. - - `generate` is now **append-only**: it refuses to write when the rebuild would - drop a committed vector. It previously rebuilt `vectors` from the legacy set - alone, so a bin vector with no legacy base was erased silently — and because - `verify`'s orphan FAIL names `generate` as the remedy, the documented repair - step completed the data loss. Reproduced end to end: dropping legacy - `width_boundary_bin16` (the fleet's only bin16 coverage) left `generate` - reporting success on a fixture two vectors smaller, with CI green. - - `--require-extras` is rejected outside `verify` (exit 2). It was accepted and - silently ignored on `generate`, the fixture-writing path — the same - accepted-and-dropped fail-open the unrecognised-argument check closes. - - Unrecognised arguments now exit 2 instead of being dropped, closing a - fail-open in the new flag itself: `verify --require-extra` (one character - short) exited 0 with the extras legs silently off. - - The set of vectors liblz4 fails to reproduce on encode is pinned in - `LZ4_ENCODE_DIVERGENT` and asserted, so a toolchain bump that changes it - fails CI instead of quietly making the new spec section's prose wrong. -- Second expert-panel round on the remediated verifier (crypto/protocol gate - keys off current HEAD, not "a panel ran once"). Three whole-file fail-opens, - all reproduced by execution and all previously exit-0: - - **The base-vector set is now pinned in code** (`EXPECTED_BASE_VECTORS`). - Every other check iterates the fixture's own vector list and so is - structurally blind to a vector that is simply *absent*. Dropping a legacy - base **and** its `_bin` twin together — the realistic bad-merge shape, which - the orphan-twin refusal does not cover — netted to zero in `generate`'s - append-only diff: `verify` reported "all 6 vector pairs verified" and - `generate` wrote the 12-vector fixture, both exit 0. It also silently - disarmed `LZ4_ENCODE_DIVERGENT`, since the divergent vector was no longer - iterated. Same lesson as `original_size`/`input_size` one level up: a name - list derived from the artifact under test pins nothing. - - **The fixture's declared `limits` block is now compared against the spec's - Security Limits table.** SDKs read their bounds from that block and nothing - pinned it either way, so a fixture rewriting `max_uncompressed_size` to `1` - verified green while handing every downstream reader a wrong bound. - - **A declared-divergent vector's `compressed_data` is now byte-pinned.** - `assert diverges == (name in LZ4_ENCODE_DIVERGENT)` is a one-bit check that - any other valid LZ4 block satisfies, so re-pinning `large_compressible` to - an unrelated (valid, correctly-decompressing) block passed both CI legs. The - byte-pin sits outside the optional-deps gate, so the one vector this section - exists to document is enforced on the stdlib leg too — it has no - canonical-writer check anywhere else in the fleet. - - `tools/test_wire_format_reference.py` gains mutation cases for all three, - each verified non-vacuous by deleting the guard and confirming the case - fails. Its own invocations that can reach `generate` now run against a - scratch mirror rather than the repo's sha256-pinned fixture — with the - guard regressed, the suite (CI's first step) rewrote the vendored artifact. - Exit-code-only assertions gained guard-marker checks, because python itself - exits 2 on a bad script path and 1 on a traceback, which made an - exit-code-only case pass vacuously. - - Fixture-shape rejections now name the offending vector instead of exiting - via a bare traceback. -- [`spec/wire-format.md`](spec/wire-format.md) corrections from the same panel: - the "MUST NOT byte-compare a writer's compressor output" rule is scoped to - **non-canonical** writers — unscoped, it forbade the `cachekit-core` re-encode - assertions that the very next paragraph relies on as the enforcement - mechanism, i.e. the fleet's only `lz4_flex` drift detector. The claim that - cachekit-core enforces canonical-writer reproducibility is now scoped to the - vectors that repo actually vendors: core pins `version == "1.1.0"`, so - `width_boundary_bin16` (added at 1.1.1) has no **canonical-writer - (`lz4_flex`) compressed-byte** check anywhere in the fleet, and its pinned - xxh3-64 checksum is recomputed nowhere. The earlier phrasing — "no - encode-side check anywhere" — was too broad and is corrected: this repo's - verifier does assert that vector's legacy and bin re-encode byte-identity on - every run, and liblz4 reproduces its compressed bytes on the optional leg. The - spec also now names what re-vendoring 1.1.1 into cachekit-core actually - requires: bump `FIXTURE_SHA256`, bump the version pin, **and** relax - `assert_eq!(twin_bytes[1], 0xc4)` to accept `0xc5` — that assertion demands - every twin be bin8, and `width_boundary_bin16_bin` is bin16, so a drop-in - re-vendor fails it. A remedy that fails on contact leaves the gap open longer. - -### Interop v2 — compressed-values profile (DRAFT) - -- New [`spec/interop-v2.md`](spec/interop-v2.md) (LAB-1135, protocol#52): - opt-in successor mode restoring the 2025-11-14 RFC's descoped - compressed+encrypted cross-SDK values. Values wrap in a `0xC1 0x02` + - msgpack `[method, original_size, payload:bin]` container (LZ4 block or - uncompressed), carried **inside** AES-256-GCM with a constant - four-component AAD reusing the frozen `"True"` token — deterministic - pre-AAD mode discrimination by configuration, no sniff-and-retry, and - cryptographic v1/v2 separation (cross-mode reads fail authentication). - Ships with a stdlib-only reference generator - ([`tools/interop-v2-reference.py`](tools/interop-v2-reference.py), - including a pure-Python LZ4 block codec), a zero-dependency independent - JS cross-check ([`tools/interop-v2-crosscheck.mjs`](tools/interop-v2-crosscheck.mjs)), - and [`test-vectors/interop-v2.json`](test-vectors/interop-v2.json) - (compressed, uncompressed, and non-canonical-widths round-trips, - compressed+encrypted round-trip, 16 structural + 2 cryptographic - must-reject vectors). Security limits - reuse the wire-format constants (512 MiB / 1000:1, enforced before - decompression); the CRIME/BREACH verdict is recorded in-spec (in threat - model, accepted with normative mitigations); the legacy array-of-ints - payload leniency is explicitly **not** inherited. Interop/v1 is - byte-for-byte untouched — its vectors and tools run unchanged beside the - new ones in CI. Status DRAFT until the vectors run in cachekit-py/ts/rs CI. - -### SDK Feature Matrix - -- Consolidated ten conflicting open matrix PRs into one code-verified end-state - (LAB-1400), regenerated from current SDK code rather than from the stale PR - diffs. Cells that **reversed** — check these if you built on them: key - rotation (py/rs ✅ → ❌ fleet-wide; `rotate_key()` is a `NotImplemented` - stub, and cachekit-py's importable PyO3 `KeyRotationState` succeeds while - rotating nothing), Rust `::secure` preset and Rust sync support (both ✅ → - never existed), Builder API (py/ts ✅ → ❌), hardware-acceleration detection - (rs ✅ → not re-exported; ts N/A → ❌), TypeScript Arrow (🔜 → ❌), and - Python's encrypted read path (documented fail-closed → **fail-open by - default**), and `cache.secure.wrap()` in TypeScript (implied encryption → no - guarantee at all; LAB-513, CWE-311). New rows: Retry, Graceful degradation, - Cross-instance L1 invalidation (LAB-520), client-L1 stale-while-revalidate - (LAB-728), Orjson serializer, tamper/wrong-key failure mode, `secure`-API - enforcement, plus an Observability section (LAB-275). Supersedes protocol#25, - #28, #29, #31, #32, #33, #35, #37, #40, #43 — per-PR fold verdicts below. - -- **Every version-keyed claim re-verified against published artifacts**, after a - review found the first pass had introduced two new false cells of - the very class it was fixing. `cachekit-rs` 0.6.0 published 74 minutes before - that pass's final commit, so six Rust reliability cells shipped marked 🚧 - unreleased when the tier was in fact released and **on by default**; and - "cachekit-ts ships the protocol-1.1 `bin` flip as of 0.1.5" was false — - published `@cachekit-io/cachekit@0.1.5` carries dependency pins byte-identical - to 0.1.4's, on `cachekit-core-ts@0.1.2` (native addons embed core **0.2.0**) - and `cachekit-core-wasm@0.1.1` (core **0.3.0**), so TypeScript emits legacy on - both paths. The matrix now carries a per-artifact rollout table with the - embedded-core evidence, and the method is recorded in - [decisions/matrix-version-verification.md](decisions/matrix-version-verification.md): - registry metadata establishes which artifact is current, and where an embedded - dependency decides the claim the `.crate`/`.tgz` is opened. Versions in the SDK - Overview are now floors (`X+`), enforced by - [`tools/check-version-floors.py`](tools/check-version-floors.py) in `verify.yml` - — four failures of this one mechanical class in six weeks. - -- Footnote namespace repaired: markers `¹`–`⁴` were each defined **twice** with - unrelated content (Cache Backends and Protocol Compliance), so half the - evidence pointers in the file resolved to the wrong note — including the - "version cells are floors" note. The Protocol Compliance block is now `¹⁴`–`¹⁷` - and every marker is defined exactly once. - -#### Per-PR fold verdicts (LAB-1400) - -| PR | Ticket | Verdict | -| :--- | :--- | :--- | -| [#25](https://github.com/cachekit-io/protocol/pull/25) | LAB-423 | Incorporated — `spec/wire-format.md` lacked the CI-enforcement note; both enforcement points now named | -| [#28](https://github.com/cachekit-io/protocol/pull/28) | LAB-274 | Incorporated incl. the intent-preset semantics table; rejected its stale "rs has no circuit breaker" line | -| [#29](https://github.com/cachekit-io/protocol/pull/29) | LAB-275 | Partly incorporated (key rotation, hardware accel, Observability, serializer rows, MSRV 1.85); versions / ts-Workers / rs-stampede / interop claims rejected as stale | -| [#31](https://github.com/cachekit-io/protocol/pull/31) | LAB-520 | Incorporated as-is | -| [#32](https://github.com/cachekit-io/protocol/pull/32) | LAB-426 | Incorporated — rs Workers locking + TTL was still missing from `main` | -| [#33](https://github.com/cachekit-io/protocol/pull/33) | LAB-427 | Already on `main`; would have reintroduced a stale rs-Redis-lock ❌ | -| [#35](https://github.com/cachekit-io/protocol/pull/35) | LAB-518 | Incorporated; rejected its "backpressure stays ❌" line (LAB-729) | -| [#37](https://github.com/cachekit-io/protocol/pull/37) | LAB-430 | Already on `main`; same stale-cell problem as #33 | -| [#40](https://github.com/cachekit-io/protocol/pull/40) | LAB-751 | Incorporated — `main` still claimed "SWR forced off" on Workers | -| [#43](https://github.com/cachekit-io/protocol/pull/43) | LAB-728 | Incorporated; extended the py cell, which understated its gate (needs an explicit `ttl=`) | -| [#17](https://github.com/cachekit-io/protocol/pull/17) | — | Out of scope, left open | - -### Specs - -- SaaS API aligned with the deployed server (LAB-677). `DELETE /v1/cache/{key}` - is idempotent — `200 {"success": true}` whether or not the key existed, never - `404`. `GET /v1/cache/health` returns `{"status","cache_entries","active_locks"}`, - not `{"version"}`. Omitting `X-CacheKit-TTL` stores a **no-expiry** entry (no - tenant-default TTL exists); the no-expiry contract is now written down — - permanently fresh while present, never age-evicted, and eligible for whatever - capacity eviction the deployment applies — "no expiry" is not a durability - guarantee; a deployment needing a ceiling provisions a quota. `PATCH /v1/cache/{key}/ttl` never `404`s (no-op on - an absent key). `404` on `GET /v1/cache/{key}/ttl` is classified "Key - absent", not a cache miss; a no-expiry entry returns `200 {"ttl": null}` - there, not `404` (see the `X-CacheKit-Fresh-For` entries above). Authentication: accepted key prefixes are `ck_sdk_` / `ck_api_` / - `ck_live_` (`ck_test_` removed — it never authenticated); `X-CacheKit-L1-Status` - moved to Required Headers as mandatory for `ck_sdk_` keys (`400` otherwise); - the `ns:`/`nsapi:` write-space split documented — each class may also mutate - unprefixed (`default`) keys, which are a shared write space; `OPTIONS` (any - path) documented as the CORS-preflight authentication exception. Stale-while- - revalidate marked shipped; the never-emitted `201` dropped from the status - table. **`HEAD` on a missing key stays `404`** (RFC 9110 §9.3.2) — the deployed - server's `200` is recorded as a known server deviation, not adopted. SDKs MUST - NOT work around it (`exists()` keeps branching on status); callers needing an - existence check against the deployed server MAY issue a raw `GET` themselves - until it is corrected — its `404` may trail a `PUT` made through another edge - instance by up to the ~5 s negative-cache window - ([fallback details](spec/saas-api.md#head-v1cachekey)). - -- StorageEnvelope `compressed_data` canonical encoding flipped from MessagePack - array-of-ints to `bin` (LAB-783 / - [cachekit-core#54](https://github.com/cachekit-io/cachekit-core/issues/54)): - protocol 1.1+ writers MUST emit `bin`; readers MUST accept both encodings - permanently. **Not a breaking change** — dual-read is mutual in both directions - under rmp-serde, toolchain-verified; no version field or discriminator. - `checksum` stays array-of-ints; `format` untouched. Tiny envelopes may grow - ≤ +1 B; incompressible payloads shrink ~35%. Rationale, evidence, and rollout - order in [decisions/envelope-bin-encoding.md](decisions/envelope-bin-encoding.md). - -- Key rotation specified (not yet implemented, LAB-516): client-side keyring — - one forward-only current key plus ≤3 decrypt-only master keys; fingerprint-based - key selection where per-entry identity exists (cachekit-py frames), sequential - same-AAD attempts elsewhere; no wire change. Retires the never-written 32-byte - `RotationAwareHeader` from the spec. Rationale, rejected options, and operator - runbooks in [decisions/key-rotation.md](decisions/key-rotation.md). - -- Interop mode promoted from draft to specified (interop/v1): flat canonical argument - array (named→positional binding), number canonicalization (integral float64 → int, - the only rule implementable in JS), code-point map-key ordering, encoded-byte set - ordering (with post-normalization dedupe), bit-deterministic datetime rule (floor - toward −∞, pre-epoch supported), full-string segment validation, canonical - (shortest-form) MessagePack encoding, plain-MessagePack value format, unchanged - AAD v0x03. Design rationale recorded in the spec's Design Decisions section. - ([#1](https://github.com/cachekit-io/protocol/issues/1)) - -### Test Vectors - -- **`verify` now enforces the python-frame default-path twin claim** (LAB-3967, - follow-up to LAB-1203). The `bin` twin in - `test-vectors/python-frame.json` carries a new operator-owned - `"twin_of": "default_saas_write_msgpack_bytestorage"` field, and - `python-frame-reference.py verify` hard-fails any declared twin that differs - from its base beyond envelope encoding — `value_json`, frame-prefix bytes - (compared as bytes, so a header key reorder is caught), `compressed_data_hex`, - `checksum_hex`, `original_size`, `format`, `inner_msgpack_hex` — or names a - vector that does not exist. Until now that check ran only at `generate` time, - as a stderr warning nobody in CI reads. The gate is keyed on the declaration, - not on the bytes, because from the bytes alone "the wheel drifted" and "the - protocol legitimately moved while the legacy vector stayed frozen" are - indistinguishable — mirroring the byte compare into CI would have moved the - LAB-1203 generator deadlock one level up. `generate` keeps warning (never - raising) and now names the two exits: fix the wheel/codec, or drop `twin_of` - in the same commit as the regenerated bytes. `_upsert` carries `twin_of` - across rebuilds and never adds or drops it; the generator's `_bin` - description no longer makes the twin claim in prose (the field is the claim), - so the exit survives the next `generate`. A declaration that cannot hold — - pointing at itself, at a same-encoding copy, at an envelope-less or - partial-envelope vector, or into a fixture with duplicate vector names — - fails rather than passing vacuously (a missing envelope field is a `FAIL` - line, never a traceback). Pinned by `tools/test_python_frame_reference.py` (mutation suite - over the committed fixture). **Fixture sha256 changes** — JSON metadata only - (`twin_of` added, `_bin` description reworded); every `frame_hex`, - `expected_payload_hex`, `expected_header` and `payload_envelope` byte is - unchanged: `d8a3756a6814971f5a2c3e6573908f60789056ecf4a2f59c28bad525745a4b6c` → - `f43eb733ccccdd9b75e95f9695719733ec78c7e31f06d8db42f0cd12b5160ac7` (no SDK - vendored this fixture when this landed, so nothing downstream re-pins). - -- `tools/python-frame-reference.py generate` now **upserts by vector name** - (LAB-1203): it rebuilds only the vectors the installed `cachekit` wheel can - reproduce and leaves every other committed vector byte-untouched, so dropping - a committed vector is structurally impossible — which deletes the LAB-903 - drop-refusal guard and both wheel-direction refusals, and folds the - append-only `generate-bin-twin` mode into `generate` (a protocol 1.1 wheel - rebuilds the `_bin` twin; the twin check is keyed on `twin_of` — see the - entry above). At `generate` time that check is a stderr - **warning**, not a hard failure: - the legacy wheel is gone from every installable release, so `legacy` can - never be regenerated — a `_require()` there would permanently deadlock - `generate` the first time the default write path legitimately changes for - reasons other than the encoding flip. A human reviews the reported diff and - decides whether it's a codec/wheel regression or a genuine protocol - evolution; `generate` itself cannot tell the two apart. Pinned by - `tools/test_python_frame_reference.py` (new; wired into `verify.yml` - alongside the frame reference verify step), which asserts the divergence - path warns and does not raise. - The ByteStorage envelope codec is no longer reimplemented there: encode/decode - come from `tools/wire-format-reference.py`, the one shared implementation of - the encoding these fixtures pin. Rewritten vectors carry per-vector - `generator` provenance (and the top-level provenance flips to an explicit - "mixed provenance" statement the first time a previously-unstamped vector is - rewritten); `test-vectors/python-frame.json` is byte-unchanged by this - refactor, and a no-op `generate` never rewrites the file. The stdlib `verify` - leg got strictly stronger: it now fully decodes each - `payload_envelope` via the shared codec (enforcing the protocol 1.1 flip - exclusions — checksum stays an array of 8 integers, format stays fixstr), - requires the envelope to re-encode byte-identically (pinning the canonical - rmp_serde shortest-form encoding, including the outer fixarray(4) marker), - and pins the declared `compressed_data_hex`/`checksum_hex`/`original_size`/ - `format` fields against the actual envelope bytes; the generate-time twin - proof now compares frame prefixes at the byte level, not as parsed JSON. - -- 7 legacy/`bin` vector pairs in `test-vectors/wire-format.json` (append-only; - legacy vectors are retained forever as legacy-read proof; fixture - 1.0.0 → 1.1.1). The original six `bin` twins were generated by the stdlib-only - `tools/wire-format-reference.py` and byte-verified against rmp-serde output; - `verify` now runs in CI (stdlib pass + `msgpack` third-encoder conformance) — - the wire-format fixture's first protocol-side CI verification. - -- 33 interop key vectors (including every `*16`-tier MessagePack width boundary), - 4 value vectors, 1 AAD vector, 1 full HKDF→AES-256-GCM encryption round-trip - vector, and 9 must-reject error vectors (`test-vectors/interop-mode.json`). - Generated by a stdlib-only Python reference implementation - (`tools/interop-reference.py`), byte-verified by an independent JavaScript encoder - using `@noble/hashes` (`tools/interop-crosscheck.mjs`), and decrypt-verified via - Node WebCrypto; both checks run in CI (`.github/workflows/verify.yml`). + ## [1.0.0] - 2026-03-28 diff --git a/changelog.d/20260328_unreleased-since-1.0.0.md b/changelog.d/20260328_unreleased-since-1.0.0.md new file mode 100644 index 0000000..193193f --- /dev/null +++ b/changelog.d/20260328_unreleased-since-1.0.0.md @@ -0,0 +1,510 @@ +### Interop mode — `ns` and `nsapi` are reserved namespaces (LAB-5876) + +- [`spec/interop-mode.md` → Segment grammar](spec/interop-mode.md#segment-grammar): + `namespace` MUST NOT be `ns` or `nsapi`. The segment pattern admitted both, but the + resulting key starts `ns:` / `nsapi:`, which the server parses as namespace-prefixed + ([cache-key-format.md → Server-Side Requirements](spec/cache-key-format.md#server-side-requirements)): + rejected when the operation contains `.`, otherwise scoped to a namespace named after + the operation. The reservation is exact-match and namespace-only; `ns` and `nsapi` + stay valid operations. SDKs reject a reserved namespace at decoration / registration + time, on every backend. **Breaking for any deployment that uses namespace `ns` or + `nsapi`, on any backend:** it now raises at startup; migrate by renaming the namespace + (a full cache miss for that namespace). +- [`test-vectors/interop-mode.json`](test-vectors/interop-mode.json) 1.1.0: two error + vectors (`reject_reserved_namespace_ns`, `reject_reserved_namespace_nsapi`) and one key + vector (`reservation_scope`: namespace `nsapix`, operation `nsapi`) that pins the + reservation as namespace-only and exact-match. Counts: 34 key, 11 error. +- SaaS Considerations no longer calls the grammar a strict subset of what the server + accepts: the grammar admits `..` inside a segment, which the server rejects. +- SDK feature matrix: the "Test vectors in CI" cells note that fixture 1.1.0 is not yet + in any released SDK, linking the SDK PRs that vendor it. + `tools/interop-reference.py` builds every key vector through its validating + `interop_key`; `tools/interop-crosscheck.mjs` checks the segment grammar on key vectors + as well as error vectors, with the reserved names hard-coded rather than read from the + fixture. + +### Wire format — vendored-fixture coverage note corrected (LAB-1750) + +- [`spec/wire-format.md`](spec/wire-format.md) no longer says `cachekit-core` vendors + fixture 1.1.0. It pins 1.1.1, so `width_boundary_bin16_bin` has a canonical-writer + (`lz4_flex`) compressed-byte and xxh3-64 checksum check. The section now states the rule + for anyone vendoring the fixture: derive each `*_bin` twin's expected marker from its + decoded `compressed_data` length, never assume bin8 or accept any `bin` width. + +### Encryption — default-tenant conformance vector (LAB-4666) + +- [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `default_tenant` + block (file version 1.2.0): tenant `"default"` under the main master key, one entry + (`default_tenant_interop`) sealed at the interop `single_int` key over the + `issue_example_object` plaintext, so it loads directly as an interop-mode entry. An SDK + that decrypts it through its `secure` preset with **no tenant configured** has + demonstrated [intent-presets.md § Master Key Input](spec/intent-presets.md#master-key-input) + rule 5 byte-for-byte. `tools/encryption-verify.py` checks the block (block-level derived-key fingerprint, AAD + with `"default"` as component 1, seal); append-only like the main set. +- Conformance table: Python's default-`tenant_id` row flips to ✅ (cachekit-py LAB-4666 — + implicit tenant is the literal `"default"`; the persisted per-host deployment UUID is + gone, explicit `deployment_uuid` / `CACHEKIT_DEPLOYMENT_UUID` remain as the override). + +### SaaS API + +- **`X-CacheKit-Fresh-For` remaining-freshness response header (LAB-557).** + `GET /v1/cache/{key}` `200 OK` responses carry the entry's remaining freshness + in whole seconds, so SDK local caches (L1) bound backfill to + `min(local_ttl, fresh_for)` instead of restarting the freshness clock at + time-of-read — which let an entry read near the end of its window be served + locally past `fresh_until` (and, with a stale-grace window, past `evict_at`). + Additive and backward compatible: an absent header means legacy behaviour on + both sides. Spec: [saas-api.md → Remaining Freshness](spec/saas-api.md#remaining-freshness). + - **Emission.** Sent on every `GET` `200 OK` for a bounded entry; `0` on + stale-window responses (and legal on `fresh` in the final sub-second); + omitted for no-expiry entries and by pre-signal servers — both mean "no + server-side bound", same SDK action. Never sent on `HEAD`, and a `HEAD` + response MUST NOT create or extend a local bound. + - **Re-serving tiers** decay the value and MUST NOT re-stamp it; they may omit + it only on positive knowledge of no expiry, and emit `0` otherwise; a + positive value only follows a `fresh` (or unlabelled) positive source or an + accepted write with a positive effective TTL. Coherence windows of composed + tiers add up, and the revocation-propagation bound is stated as that sum + plus the local bound, transit and clock error; `evict_at` is the store's + bound, not an end-to-end one. + - **SDK consumption.** The value is a hard local service bound — once it + elapses the local copy MUST NOT be served in any form, and `0` forbids + backfill. It MUST be 1–7 ASCII digits and at most `2,592,000`; anything + else is `0`. Local caches MUST NOT backfill a `stale`-labelled response at + all. Local deadlines SHOULD use a clock that counts across suspend. +- **`Cache-Control: no-store` and `Vary: Authorization` on every response.** + The cache key carries no tenant, so byte-identical URLs across tenants made + heuristic HTTP caching (RFC 9111 §4.2.2) a cross-tenant read. CacheKit-operated + caching tiers MUST partition by tenant. +- **Effective TTL.** A write's effective TTL is `X-CacheKit-TTL` when present, + otherwise the deprecated `X-TTL`; a write stores a no-expiry entry only when + it carries neither. The no-expiry rule, the stale-window requirement and the + tier rules all read the effective TTL, so a legacy `X-TTL: 60` write is no + longer mistaken for an immortal one. +- **No-expiry follow-ons** to the contract in the LAB-677 entry below. + `GET /v1/cache/{key}/ttl` returns `200 {"ttl": null}` for a no-expiry key + (mixed-reader caveat for SDKs that predate `null`), so its `404` now means + only "key absent"; an SDK TTL read MAY still collapse both to its null value. + A revalidation `PUT` MUST re-send the TTL as well as the stale window, or it + stores a no-expiry entry. Keys whose TTL is a revocation boundary MUST be + stored with an explicit TTL. The 30-day maximum bounds a stated TTL's value + range, not an entry's storage lifetime. + +### Intent presets — canonical preset contract (LAB-514) + +- New normative [`spec/intent-presets.md`](spec/intent-presets.md): what `minimal` / + `production` / `secure` / `io` MUST configure in every SDK. Decisions: finite default + TTLs (300 / 600 / 600 / 3 600 s — Python's cache-forever default is the outlier); L1 on + for every preset and ciphertext-only on `secure` (ratifies the 2025-11-13 cachekit-py + decision cross-SDK); no MUST on integrity checksums (the storage container is + SDK-internal, protocol#11); reliability stack default-on for `production`/`secure`/`io`; + `secure` is the canonical name in every SDK (Rust's `CacheKit::encrypted` is a tracked + non-conformance, LAB-4651); the encrypted preset MUST take a hex key and fall back to + `CACHEKIT_MASTER_KEY`; **`CACHEKIT_MASTER_KEY` is a key source, not an activation + switch** — it MUST NOT turn encryption on for `minimal`/`production`/`io`, and no + constructor is exempt (Python's fleet-wide auto-detect and Rust's `from_env()` + presence-activation are the outliers); an explicit encryption option MUST encrypt + every operation or be rejected; the default `tenant_id` is `"default"` and MUST be + identical for HKDF and AAD; no SDK MAY offer a process-wide default-TTL override; + `io` takes its API key by argument **or** `CACHEKIT_API_KEY`; explicit arguments + that a preset does not support MUST be rejected, never dropped. +- Per-SDK conformance table (code-verified 2026-09-22 against `main`: py `2f7c979`, + rs `6587ce9`, ts `379847c`) with one alignment ticket per ❌; TypeScript's only ❌ is + the HKDF-vs-AAD `tenant_id` mismatch. +- [`spec/encryption.md`](spec/encryption.md#master-key) Master Key table: minimum length + corrected from 16 bytes to **32 bytes (64 hex chars)** — every SDK enforces 32 at the + configuration boundary; 16 is the HKDF core's IKM floor and was never user-facing. +- [Feature matrix](sdk-feature-matrix.md#intent-preset-semantics-parity-not-presence) + intent-preset section now links the spec; README spec index gains the row. + +### Encryption — keyring conformance vectors + status reconciliation (LAB-687) + +- [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `keyring` + block: two master keys (`k1`, `k2`) for tenant `keyring-conformance`, one entry + sealed under each, and per-vector `key_fingerprint_hex` — the fingerprint of the + HKDF-derived per-tenant encryption key, as cachekit-py stores it. Frozen names + `encrypted_with_k1` / `encrypted_with_k2`; append-only like the main set. +- [`tools/encryption-verify.py`](tools/encryption-verify.py) enforces + [`spec/encryption.md` § Key Rotation (Keyring)](spec/encryption.md#key-rotation-keyring): + `[k2, k1]` decrypts both at the declared entry, `[k2]` alone rejects the k1 + entry, and the stored fingerprint selects the derived key — a master-key + fingerprint cannot select. Entry derivation, metadata, AAD and selection run + in the stdlib lane; only the decrypt attempts need `cryptography`. +- New [`tools/test_encryption_verify.py`](tools/test_encryption_verify.py) mutation + suite runs ahead of the verifier in both CI lanes (same doctrine as the + wire-format guard): every keyring guard is proven to go red by poisoning a copy + of the fixture. Added after the LAB-687 panel found three vacuous passes in the + first revision. +- Status banners reconciled with shipped code: `spec/encryption.md` and + `decisions/key-rotation.md` no longer say "not yet implemented"; the + [feature matrix](sdk-feature-matrix.md#encryption) Key rotation row is ✅ for + Python 0.18.0+ (cachekit-py#261) and 🚧 unreleased for Rust and TypeScript — + cachekit-rs#63 and cachekit-ts#103 are merged but absent from crates.io 0.7.0 + and npm 0.1.5 (artifacts inspected 2026-09-22). +- `decisions/key-rotation.md` compromise runbook: flush at cut-over **and** again + once the last old-key writer has stopped — a deferred single flush leaves the + whole compromised corpus readable for the rollout window; the decrypt-only + list must be explicitly empty, since an omitted list falls back to + `CACHEKIT_PREVIOUS_MASTER_KEYS`. + +### SaaS API — `401` is an authoritative verdict; auth backend faults are `503` (LAB-4093) + +- [`spec/saas-api.md`](spec/saas-api.md) Error Handling: `401` is emitted only + for an authoritative denial — a missing or malformed `Authorization` header, + or the key store saying the key is unknown, revoked, or its tenant suspended or soft-deleted. + A backend fault while resolving the key (auth cache / database) is a `503` + with `Retry-After`, so SDKs retry under the existing Transient class instead + of surfacing a transient blip as "invalid API key" (Permanent, never retried). + Documents the cache-worker behaviour shipped in + [cachekit-io/saas#380](https://github.com/cachekit-io/saas/pull/380); no + SDK change — `503` already classifies as Transient in all three. + +### SDK feature matrix — TypeScript `cache.secure.wrap()` now fails closed (LAB-513) + +- [`sdk-feature-matrix.md`](sdk-feature-matrix.md): the Encryption row "Does the + `secure` API enforce encryption?" flips ❌ → ✅ for TypeScript. Both + `cache.secure.wrap()` and `cache.withExecutionContext(ctx).secure.wrap()` now + throw `ConfigurationError` at wrap time on any instance without `encryption` + configured ([cachekit-ts#123](https://github.com/cachekit-io/cachekit-ts/pull/123)); + before, both were unconditional aliases for `wrap()`, so on an instance + without configured encryption a "secure" registration stored plaintext + (CWE-311) — on an encrypted instance they always encrypted. All three SDKs + now refuse a missing key on the secure entry point — py raises at decoration + time, rs `secure()` returns `Err`, ts throws at wrap time — with no opt-in to + run the secure entry point unencrypted in any of them; ts callers who want + plaintext call `wrap()` explicitly. The "Intent-preset + semantics" warning is rewritten to the enforced contract, the "cells that + reversed" summary and the Rust builder-stub cross-reference are updated to + match, and the stale `cache-core.ts:832` / `:873` / `:486`, `cache.ts:87` and + `intents-core.ts:240` references are replaced with current ones. + + +### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) + +- LZ4 compressed bytes are **not canonical** across conforming block encoders. + [`spec/wire-format.md`](spec/wire-format.md) now states this explicitly + (new "Compressed-byte reproducibility" section, mirroring interop v2's + doctrine): `compressed_data` conformance is read-side only, a **non-canonical** + writer is never judged non-conforming for differing from the pinned bytes + (byte-comparison as a declared-divergence tripwire remains allowed), and + only the canonical writer (`lz4_flex` via `cachekit-core`) has enforced + byte-reproducibility. The `large_compressible` / `large_compressible_bin` + pair is marked **known encode-divergent, decode-verified only** under the + spec's own reference liblz4 mapping — `lz4.block.compress(store_size=False)` + emits a 14 B block where the fixture pins `lz4_flex`'s 15 B. Found by + execution during the LAB-868 panel review; resolves the trust bug of a + fixture implying a reproducibility property the reference toolchain cannot + produce. Regeneration was rejected: every envelope-using SDK compresses + through `cachekit-core`'s `lz4_flex` (`cachekit-rs` writes plain MessagePack + with no envelope — spec 'SDK Storage Containers (auto mode)'), whose CI asserts re-encode byte-identity, so + re-pinning to liblz4 output would break the canonical writer and merely swap + which compressor diverges. +- [`tools/wire-format-reference.py`](tools/wire-format-reference.py) `verify` + gains an optional `lz4` leg (the dependency was already installed in CI's + optional-deps step): liblz4 MUST decompress every pinned `compressed_data` + to the pinned input; encoder agreement is asserted only as a drift + tripwire against `LZ4_ENCODE_DIVERGENT`, never as a per-vector conformance rule. The CI invocation now passes `--require-extras` + (precedent: `encryption-verify.py --require-seal`) so a dependency drift + cannot silently turn the deeper checks off. Fixture bytes untouched + (version stays 1.1.1) — no downstream SDK re-vendors required. +- Expert-panel hardening of the same verifier (crypto/protocol gate; every item + below was reproduced by poisoning the fixture and re-run after the fix): + - `original_size` is now checked against `len(input_hex)`, not just the + co-located `input_size` field. Both declared sizes live *in* the file under + test, so a regeneration bug that inflates them drifts them together and the + old check still passed — a vector declaring 100 MB for 16 bytes of input + verified green, and liblz4 did not catch it because + `decompress(uncompressed_size=…)` sizes the output buffer rather than + asserting the length. Runs on both CI legs (stdlib and optional-deps). + - **Both** commands refuse to run under `-O`/`PYTHONOPTIMIZE`: every conformance + check is an `assert`, so an optimised `verify` reported "all 7 vector pairs + verified" against a poisoned fixture, and an optimised `generate` rewrote the + fixture with its input checks stripped. The guard is at module scope, not in + `main()`, because a CLI-only guard is bypassed by importing the module and + calling `verify()` directly — which the regression harness's `importlib` + probe does, and which is how the sibling tools load each other's codecs. + - `generate` is now **append-only**: it refuses to write when the rebuild would + drop a committed vector. It previously rebuilt `vectors` from the legacy set + alone, so a bin vector with no legacy base was erased silently — and because + `verify`'s orphan FAIL names `generate` as the remedy, the documented repair + step completed the data loss. Reproduced end to end: dropping legacy + `width_boundary_bin16` (the fleet's only bin16 coverage) left `generate` + reporting success on a fixture two vectors smaller, with CI green. + - `--require-extras` is rejected outside `verify` (exit 2). It was accepted and + silently ignored on `generate`, the fixture-writing path — the same + accepted-and-dropped fail-open the unrecognised-argument check closes. + - Unrecognised arguments now exit 2 instead of being dropped, closing a + fail-open in the new flag itself: `verify --require-extra` (one character + short) exited 0 with the extras legs silently off. + - The set of vectors liblz4 fails to reproduce on encode is pinned in + `LZ4_ENCODE_DIVERGENT` and asserted, so a toolchain bump that changes it + fails CI instead of quietly making the new spec section's prose wrong. +- Second expert-panel round on the remediated verifier (crypto/protocol gate + keys off current HEAD, not "a panel ran once"). Three whole-file fail-opens, + all reproduced by execution and all previously exit-0: + - **The base-vector set is now pinned in code** (`EXPECTED_BASE_VECTORS`). + Every other check iterates the fixture's own vector list and so is + structurally blind to a vector that is simply *absent*. Dropping a legacy + base **and** its `_bin` twin together — the realistic bad-merge shape, which + the orphan-twin refusal does not cover — netted to zero in `generate`'s + append-only diff: `verify` reported "all 6 vector pairs verified" and + `generate` wrote the 12-vector fixture, both exit 0. It also silently + disarmed `LZ4_ENCODE_DIVERGENT`, since the divergent vector was no longer + iterated. Same lesson as `original_size`/`input_size` one level up: a name + list derived from the artifact under test pins nothing. + - **The fixture's declared `limits` block is now compared against the spec's + Security Limits table.** SDKs read their bounds from that block and nothing + pinned it either way, so a fixture rewriting `max_uncompressed_size` to `1` + verified green while handing every downstream reader a wrong bound. + - **A declared-divergent vector's `compressed_data` is now byte-pinned.** + `assert diverges == (name in LZ4_ENCODE_DIVERGENT)` is a one-bit check that + any other valid LZ4 block satisfies, so re-pinning `large_compressible` to + an unrelated (valid, correctly-decompressing) block passed both CI legs. The + byte-pin sits outside the optional-deps gate, so the one vector this section + exists to document is enforced on the stdlib leg too — it has no + canonical-writer check anywhere else in the fleet. + - `tools/test_wire_format_reference.py` gains mutation cases for all three, + each verified non-vacuous by deleting the guard and confirming the case + fails. Its own invocations that can reach `generate` now run against a + scratch mirror rather than the repo's sha256-pinned fixture — with the + guard regressed, the suite (CI's first step) rewrote the vendored artifact. + Exit-code-only assertions gained guard-marker checks, because python itself + exits 2 on a bad script path and 1 on a traceback, which made an + exit-code-only case pass vacuously. + - Fixture-shape rejections now name the offending vector instead of exiting + via a bare traceback. +- [`spec/wire-format.md`](spec/wire-format.md) corrections from the same panel: + the "MUST NOT byte-compare a writer's compressor output" rule is scoped to + **non-canonical** writers — unscoped, it forbade the `cachekit-core` re-encode + assertions that the very next paragraph relies on as the enforcement + mechanism, i.e. the fleet's only `lz4_flex` drift detector. The claim that + cachekit-core enforces canonical-writer reproducibility is now scoped to the + vectors that repo actually vendors: core pins `version == "1.1.0"`, so + `width_boundary_bin16` (added at 1.1.1) has no **canonical-writer + (`lz4_flex`) compressed-byte** check anywhere in the fleet, and its pinned + xxh3-64 checksum is recomputed nowhere. The earlier phrasing — "no + encode-side check anywhere" — was too broad and is corrected: this repo's + verifier does assert that vector's legacy and bin re-encode byte-identity on + every run, and liblz4 reproduces its compressed bytes on the optional leg. The + spec also now names what re-vendoring 1.1.1 into cachekit-core actually + requires: bump `FIXTURE_SHA256`, bump the version pin, **and** relax + `assert_eq!(twin_bytes[1], 0xc4)` to accept `0xc5` — that assertion demands + every twin be bin8, and `width_boundary_bin16_bin` is bin16, so a drop-in + re-vendor fails it. A remedy that fails on contact leaves the gap open longer. + +### Interop v2 — compressed-values profile (DRAFT) + +- New [`spec/interop-v2.md`](spec/interop-v2.md) (LAB-1135, protocol#52): + opt-in successor mode restoring the 2025-11-14 RFC's descoped + compressed+encrypted cross-SDK values. Values wrap in a `0xC1 0x02` + + msgpack `[method, original_size, payload:bin]` container (LZ4 block or + uncompressed), carried **inside** AES-256-GCM with a constant + four-component AAD reusing the frozen `"True"` token — deterministic + pre-AAD mode discrimination by configuration, no sniff-and-retry, and + cryptographic v1/v2 separation (cross-mode reads fail authentication). + Ships with a stdlib-only reference generator + ([`tools/interop-v2-reference.py`](tools/interop-v2-reference.py), + including a pure-Python LZ4 block codec), a zero-dependency independent + JS cross-check ([`tools/interop-v2-crosscheck.mjs`](tools/interop-v2-crosscheck.mjs)), + and [`test-vectors/interop-v2.json`](test-vectors/interop-v2.json) + (compressed, uncompressed, and non-canonical-widths round-trips, + compressed+encrypted round-trip, 16 structural + 2 cryptographic + must-reject vectors). Security limits + reuse the wire-format constants (512 MiB / 1000:1, enforced before + decompression); the CRIME/BREACH verdict is recorded in-spec (in threat + model, accepted with normative mitigations); the legacy array-of-ints + payload leniency is explicitly **not** inherited. Interop/v1 is + byte-for-byte untouched — its vectors and tools run unchanged beside the + new ones in CI. Status DRAFT until the vectors run in cachekit-py/ts/rs CI. + +### SDK Feature Matrix + +- Consolidated ten conflicting open matrix PRs into one code-verified end-state + (LAB-1400), regenerated from current SDK code rather than from the stale PR + diffs. Cells that **reversed** — check these if you built on them: key + rotation (py/rs ✅ → ❌ fleet-wide; `rotate_key()` is a `NotImplemented` + stub, and cachekit-py's importable PyO3 `KeyRotationState` succeeds while + rotating nothing), Rust `::secure` preset and Rust sync support (both ✅ → + never existed), Builder API (py/ts ✅ → ❌), hardware-acceleration detection + (rs ✅ → not re-exported; ts N/A → ❌), TypeScript Arrow (🔜 → ❌), and + Python's encrypted read path (documented fail-closed → **fail-open by + default**), and `cache.secure.wrap()` in TypeScript (implied encryption → no + guarantee at all; LAB-513, CWE-311). New rows: Retry, Graceful degradation, + Cross-instance L1 invalidation (LAB-520), client-L1 stale-while-revalidate + (LAB-728), Orjson serializer, tamper/wrong-key failure mode, `secure`-API + enforcement, plus an Observability section (LAB-275). Supersedes protocol#25, + #28, #29, #31, #32, #33, #35, #37, #40, #43 — per-PR fold verdicts below. + +- **Every version-keyed claim re-verified against published artifacts**, after a + review found the first pass had introduced two new false cells of + the very class it was fixing. `cachekit-rs` 0.6.0 published 74 minutes before + that pass's final commit, so six Rust reliability cells shipped marked 🚧 + unreleased when the tier was in fact released and **on by default**; and + "cachekit-ts ships the protocol-1.1 `bin` flip as of 0.1.5" was false — + published `@cachekit-io/cachekit@0.1.5` carries dependency pins byte-identical + to 0.1.4's, on `cachekit-core-ts@0.1.2` (native addons embed core **0.2.0**) + and `cachekit-core-wasm@0.1.1` (core **0.3.0**), so TypeScript emits legacy on + both paths. The matrix now carries a per-artifact rollout table with the + embedded-core evidence, and the method is recorded in + [decisions/matrix-version-verification.md](decisions/matrix-version-verification.md): + registry metadata establishes which artifact is current, and where an embedded + dependency decides the claim the `.crate`/`.tgz` is opened. Versions in the SDK + Overview are now floors (`X+`), enforced by + [`tools/check-version-floors.py`](tools/check-version-floors.py) in `verify.yml` + — four failures of this one mechanical class in six weeks. + +- Footnote namespace repaired: markers `¹`–`⁴` were each defined **twice** with + unrelated content (Cache Backends and Protocol Compliance), so half the + evidence pointers in the file resolved to the wrong note — including the + "version cells are floors" note. The Protocol Compliance block is now `¹⁴`–`¹⁷` + and every marker is defined exactly once. + +#### Per-PR fold verdicts (LAB-1400) + +| PR | Ticket | Verdict | +| :--- | :--- | :--- | +| [#25](https://github.com/cachekit-io/protocol/pull/25) | LAB-423 | Incorporated — `spec/wire-format.md` lacked the CI-enforcement note; both enforcement points now named | +| [#28](https://github.com/cachekit-io/protocol/pull/28) | LAB-274 | Incorporated incl. the intent-preset semantics table; rejected its stale "rs has no circuit breaker" line | +| [#29](https://github.com/cachekit-io/protocol/pull/29) | LAB-275 | Partly incorporated (key rotation, hardware accel, Observability, serializer rows, MSRV 1.85); versions / ts-Workers / rs-stampede / interop claims rejected as stale | +| [#31](https://github.com/cachekit-io/protocol/pull/31) | LAB-520 | Incorporated as-is | +| [#32](https://github.com/cachekit-io/protocol/pull/32) | LAB-426 | Incorporated — rs Workers locking + TTL was still missing from `main` | +| [#33](https://github.com/cachekit-io/protocol/pull/33) | LAB-427 | Already on `main`; would have reintroduced a stale rs-Redis-lock ❌ | +| [#35](https://github.com/cachekit-io/protocol/pull/35) | LAB-518 | Incorporated; rejected its "backpressure stays ❌" line (LAB-729) | +| [#37](https://github.com/cachekit-io/protocol/pull/37) | LAB-430 | Already on `main`; same stale-cell problem as #33 | +| [#40](https://github.com/cachekit-io/protocol/pull/40) | LAB-751 | Incorporated — `main` still claimed "SWR forced off" on Workers | +| [#43](https://github.com/cachekit-io/protocol/pull/43) | LAB-728 | Incorporated; extended the py cell, which understated its gate (needs an explicit `ttl=`) | +| [#17](https://github.com/cachekit-io/protocol/pull/17) | — | Out of scope, left open | + +### Specs + +- SaaS API aligned with the deployed server (LAB-677). `DELETE /v1/cache/{key}` + is idempotent — `200 {"success": true}` whether or not the key existed, never + `404`. `GET /v1/cache/health` returns `{"status","cache_entries","active_locks"}`, + not `{"version"}`. Omitting `X-CacheKit-TTL` stores a **no-expiry** entry (no + tenant-default TTL exists); the no-expiry contract is now written down — + permanently fresh while present, never age-evicted, and eligible for whatever + capacity eviction the deployment applies — "no expiry" is not a durability + guarantee; a deployment needing a ceiling provisions a quota. `PATCH /v1/cache/{key}/ttl` never `404`s (no-op on + an absent key). `404` on `GET /v1/cache/{key}/ttl` is classified "Key + absent", not a cache miss; a no-expiry entry returns `200 {"ttl": null}` + there, not `404` (see the `X-CacheKit-Fresh-For` entries above). Authentication: accepted key prefixes are `ck_sdk_` / `ck_api_` / + `ck_live_` (`ck_test_` removed — it never authenticated); `X-CacheKit-L1-Status` + moved to Required Headers as mandatory for `ck_sdk_` keys (`400` otherwise); + the `ns:`/`nsapi:` write-space split documented — each class may also mutate + unprefixed (`default`) keys, which are a shared write space; `OPTIONS` (any + path) documented as the CORS-preflight authentication exception. Stale-while- + revalidate marked shipped; the never-emitted `201` dropped from the status + table. **`HEAD` on a missing key stays `404`** (RFC 9110 §9.3.2) — the deployed + server's `200` is recorded as a known server deviation, not adopted. SDKs MUST + NOT work around it (`exists()` keeps branching on status); callers needing an + existence check against the deployed server MAY issue a raw `GET` themselves + until it is corrected — its `404` may trail a `PUT` made through another edge + instance by up to the ~5 s negative-cache window + ([fallback details](spec/saas-api.md#head-v1cachekey)). + +- StorageEnvelope `compressed_data` canonical encoding flipped from MessagePack + array-of-ints to `bin` (LAB-783 / + [cachekit-core#54](https://github.com/cachekit-io/cachekit-core/issues/54)): + protocol 1.1+ writers MUST emit `bin`; readers MUST accept both encodings + permanently. **Not a breaking change** — dual-read is mutual in both directions + under rmp-serde, toolchain-verified; no version field or discriminator. + `checksum` stays array-of-ints; `format` untouched. Tiny envelopes may grow + ≤ +1 B; incompressible payloads shrink ~35%. Rationale, evidence, and rollout + order in [decisions/envelope-bin-encoding.md](decisions/envelope-bin-encoding.md). + +- Key rotation specified (not yet implemented, LAB-516): client-side keyring — + one forward-only current key plus ≤3 decrypt-only master keys; fingerprint-based + key selection where per-entry identity exists (cachekit-py frames), sequential + same-AAD attempts elsewhere; no wire change. Retires the never-written 32-byte + `RotationAwareHeader` from the spec. Rationale, rejected options, and operator + runbooks in [decisions/key-rotation.md](decisions/key-rotation.md). + +- Interop mode promoted from draft to specified (interop/v1): flat canonical argument + array (named→positional binding), number canonicalization (integral float64 → int, + the only rule implementable in JS), code-point map-key ordering, encoded-byte set + ordering (with post-normalization dedupe), bit-deterministic datetime rule (floor + toward −∞, pre-epoch supported), full-string segment validation, canonical + (shortest-form) MessagePack encoding, plain-MessagePack value format, unchanged + AAD v0x03. Design rationale recorded in the spec's Design Decisions section. + ([#1](https://github.com/cachekit-io/protocol/issues/1)) + +### Test Vectors + +- **`verify` now enforces the python-frame default-path twin claim** (LAB-3967, + follow-up to LAB-1203). The `bin` twin in + `test-vectors/python-frame.json` carries a new operator-owned + `"twin_of": "default_saas_write_msgpack_bytestorage"` field, and + `python-frame-reference.py verify` hard-fails any declared twin that differs + from its base beyond envelope encoding — `value_json`, frame-prefix bytes + (compared as bytes, so a header key reorder is caught), `compressed_data_hex`, + `checksum_hex`, `original_size`, `format`, `inner_msgpack_hex` — or names a + vector that does not exist. Until now that check ran only at `generate` time, + as a stderr warning nobody in CI reads. The gate is keyed on the declaration, + not on the bytes, because from the bytes alone "the wheel drifted" and "the + protocol legitimately moved while the legacy vector stayed frozen" are + indistinguishable — mirroring the byte compare into CI would have moved the + LAB-1203 generator deadlock one level up. `generate` keeps warning (never + raising) and now names the two exits: fix the wheel/codec, or drop `twin_of` + in the same commit as the regenerated bytes. `_upsert` carries `twin_of` + across rebuilds and never adds or drops it; the generator's `_bin` + description no longer makes the twin claim in prose (the field is the claim), + so the exit survives the next `generate`. A declaration that cannot hold — + pointing at itself, at a same-encoding copy, at an envelope-less or + partial-envelope vector, or into a fixture with duplicate vector names — + fails rather than passing vacuously (a missing envelope field is a `FAIL` + line, never a traceback). Pinned by `tools/test_python_frame_reference.py` (mutation suite + over the committed fixture). **Fixture sha256 changes** — JSON metadata only + (`twin_of` added, `_bin` description reworded); every `frame_hex`, + `expected_payload_hex`, `expected_header` and `payload_envelope` byte is + unchanged: `d8a3756a6814971f5a2c3e6573908f60789056ecf4a2f59c28bad525745a4b6c` → + `f43eb733ccccdd9b75e95f9695719733ec78c7e31f06d8db42f0cd12b5160ac7` (no SDK + vendored this fixture when this landed, so nothing downstream re-pins). + +- `tools/python-frame-reference.py generate` now **upserts by vector name** + (LAB-1203): it rebuilds only the vectors the installed `cachekit` wheel can + reproduce and leaves every other committed vector byte-untouched, so dropping + a committed vector is structurally impossible — which deletes the LAB-903 + drop-refusal guard and both wheel-direction refusals, and folds the + append-only `generate-bin-twin` mode into `generate` (a protocol 1.1 wheel + rebuilds the `_bin` twin; the twin check is keyed on `twin_of` — see the + entry above). At `generate` time that check is a stderr + **warning**, not a hard failure: + the legacy wheel is gone from every installable release, so `legacy` can + never be regenerated — a `_require()` there would permanently deadlock + `generate` the first time the default write path legitimately changes for + reasons other than the encoding flip. A human reviews the reported diff and + decides whether it's a codec/wheel regression or a genuine protocol + evolution; `generate` itself cannot tell the two apart. Pinned by + `tools/test_python_frame_reference.py` (new; wired into `verify.yml` + alongside the frame reference verify step), which asserts the divergence + path warns and does not raise. + The ByteStorage envelope codec is no longer reimplemented there: encode/decode + come from `tools/wire-format-reference.py`, the one shared implementation of + the encoding these fixtures pin. Rewritten vectors carry per-vector + `generator` provenance (and the top-level provenance flips to an explicit + "mixed provenance" statement the first time a previously-unstamped vector is + rewritten); `test-vectors/python-frame.json` is byte-unchanged by this + refactor, and a no-op `generate` never rewrites the file. The stdlib `verify` + leg got strictly stronger: it now fully decodes each + `payload_envelope` via the shared codec (enforcing the protocol 1.1 flip + exclusions — checksum stays an array of 8 integers, format stays fixstr), + requires the envelope to re-encode byte-identically (pinning the canonical + rmp_serde shortest-form encoding, including the outer fixarray(4) marker), + and pins the declared `compressed_data_hex`/`checksum_hex`/`original_size`/ + `format` fields against the actual envelope bytes; the generate-time twin + proof now compares frame prefixes at the byte level, not as parsed JSON. + +- 7 legacy/`bin` vector pairs in `test-vectors/wire-format.json` (append-only; + legacy vectors are retained forever as legacy-read proof; fixture + 1.0.0 → 1.1.1). The original six `bin` twins were generated by the stdlib-only + `tools/wire-format-reference.py` and byte-verified against rmp-serde output; + `verify` now runs in CI (stdlib pass + `msgpack` third-encoder conformance) — + the wire-format fixture's first protocol-side CI verification. + +- 33 interop key vectors (including every `*16`-tier MessagePack width boundary), + 4 value vectors, 1 AAD vector, 1 full HKDF→AES-256-GCM encryption round-trip + vector, and 9 must-reject error vectors (`test-vectors/interop-mode.json`). + Generated by a stdlib-only Python reference implementation + (`tools/interop-reference.py`), byte-verified by an independent JavaScript encoder + using `@noble/hashes` (`tools/interop-crosscheck.mjs`), and decrypt-verified via + Node WebCrypto; both checks run in CI (`.github/workflows/verify.yml`). diff --git a/changelog.d/README.md b/changelog.d/README.md new file mode 100644 index 0000000..528ca17 --- /dev/null +++ b/changelog.d/README.md @@ -0,0 +1,42 @@ +# Changelog fragments + +Each unreleased change gets its own file in this directory. Pull requests do not edit +[`CHANGELOG.md`](../CHANGELOG.md): when every PR adds its entry under the same +`## [Unreleased]` heading, any two open PRs conflict. Separate files never do. CI fails a +PR that edits `CHANGELOG.md`, unless the PR is a release (see below). + +## Adding an entry + +Create `changelog.d/_.md`, for example +`changelog.d/20260929_lab-6153.md`, using the date you write it. The file holds the entry +exactly as it will appear in the changelog: a `###` heading naming the area and the change, +then bullets. + +```markdown +### Wire format — short statement of the change (LAB-1234) + +- What changed, with a link to the spec section. +- **Breaking for …:** who is affected and how to migrate, when it applies. +``` + +Files are collected in filename order, so entries appear in the order they were written, +not the order they merged. Never let one entry's meaning depend on appearing before or +after another. + +A change that needs no changelog entry does not add a file. + +## Releasing + +Releases are cut from a branch named `release/`. That prefix is the CI exemption. + +```bash +git checkout -b release/1.1.0 origin/main +python3 tools/changelog-collect.py 1.1.0 +``` + +`tools/changelog-collect.py` writes a `## [1.1.0] - ` section at the +`` marker in `CHANGELOG.md`, holding every fragment verbatim +in filename order, and deletes the collected fragments. It never regroups or rewrites +entries. This README stays, and so does the marker. Everything merged between 1.0.0 and +this directory's introduction is in `20260328_unreleased-since-1.0.0.md`, which sorts +first, so the first release's section contains it. diff --git a/tools/changelog-collect.py b/tools/changelog-collect.py new file mode 100644 index 0000000..439954d --- /dev/null +++ b/tools/changelog-collect.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Collect changelog.d/ fragments into a new CHANGELOG.md version section. + +Each fragment is inserted verbatim, in filename order, under `## [VERSION] - DATE` +at the insert marker; the collected fragments are then deleted. Nothing is parsed, +regrouped or reordered. That is the reason this exists instead of scriv: scriv reads +every `###` heading as a category, so it merged same-titled entries (`### SaaS API`) +and shuffled entry order across the release. These entries are curated normative +prose, so a verbatim concatenation is the only safe transform. + +Fails closed: a missing or repeated marker, an existing version, no fragments, or an +empty fragment is an error, and nothing is written. + +Usage: python3 tools/changelog-collect.py VERSION [--date YYYY-MM-DD] [--root PATH] +""" + +from __future__ import annotations + +import argparse +import datetime +import re +import sys +from pathlib import Path + +MARKER = "" +VERSION_RE = re.compile(r"\d+\.\d+\.\d+") + + +def collect(root: Path, version: str, date: str) -> list[Path]: + """Write the version section; return the fragments it collected (already deleted).""" + if not VERSION_RE.fullmatch(version): + raise ValueError(f"version must be MAJOR.MINOR.PATCH, got {version!r}") + changelog = root / "CHANGELOG.md" + text = changelog.read_text(encoding="utf-8") + if text.count(MARKER) != 1: + raise ValueError(f"{changelog} must contain {MARKER} exactly once, found {text.count(MARKER)}") + if re.search(rf"^## \[{re.escape(version)}\]", text, re.MULTILINE): + raise ValueError(f"{changelog} already has a {version} section") + fragments = sorted(p for p in (root / "changelog.d").glob("*.md") if p.name != "README.md") + if not fragments: + raise ValueError("no fragments in changelog.d/") + bodies = [p.read_text(encoding="utf-8").strip() for p in fragments] + empty = [p.name for p, body in zip(fragments, bodies) if not body] + if empty: + raise ValueError(f"empty fragment(s): {', '.join(empty)}") + + head, rest = text.split(MARKER) + section = f"## [{version}] - {date}\n\n" + "\n\n".join(bodies) + "\n\n" + changelog.write_text(f"{head}{MARKER}\n\n{section}{rest.lstrip(chr(10))}", encoding="utf-8") + for p in fragments: + p.unlink() + return fragments + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("version") + ap.add_argument("--date", type=datetime.date.fromisoformat, default=datetime.datetime.now(datetime.UTC).date()) + ap.add_argument("--root", type=Path, default=Path(__file__).resolve().parent.parent) + args = ap.parse_args() + try: + collected = collect(args.root, args.version, args.date.isoformat()) + except (ValueError, OSError) as e: + print(f"changelog-collect: {e}", file=sys.stderr) + return 1 + print(f"collected {len(collected)} fragment(s) into [{args.version}]:") + for p in collected: + print(f" {p.name}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/test_changelog_collect.py b/tools/test_changelog_collect.py new file mode 100644 index 0000000..ba88d63 --- /dev/null +++ b/tools/test_changelog_collect.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Tests for changelog-collect.py. + +The tool replaced scriv after scriv merged same-titled `###` entries and reordered +a release, so the properties pinned here are the ones scriv broke: every fragment +byte lands verbatim, in filename order, with duplicate headings kept apart. Each +fail-closed refusal must also leave CHANGELOG.md and the fragments untouched. + +Run: python3 tools/test_changelog_collect.py (exit 1 on any failure) +""" + +from __future__ import annotations + +import importlib.util +import re +import shutil +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +REPO = HERE.parent +_spec = importlib.util.spec_from_file_location("changelog_collect", HERE / "changelog-collect.py") +assert _spec and _spec.loader +cc = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(cc) + +BASE = f"# Changelog\n\n## [Unreleased]\n\nPointer.\n\n{cc.MARKER}\n\n## [1.0.0] - 2026-03-28\n\nInitial.\n" +FAILURES: list[str] = [] + + +def check(name: str, cond: bool) -> None: + print(f"{'ok ' if cond else 'FAIL'} {name}") + if not cond: + FAILURES.append(name) + + +def make(tmp: Path, changelog: str, fragments: dict[str, str]) -> Path: + root = Path(tempfile.mkdtemp(dir=tmp)) + (root / "changelog.d").mkdir() + (root / "changelog.d" / "README.md").write_text("readme\n") + (root / "CHANGELOG.md").write_text(changelog) + for name, body in fragments.items(): + (root / "changelog.d" / name).write_text(body) + return root + + +def refuses(tmp: Path, name: str, changelog: str, fragments: dict[str, str], version: str = "1.1.0") -> None: + root = make(tmp, changelog, fragments) + try: + cc.collect(root, version, "2026-10-01") + raised = False + except ValueError: + raised = True + untouched = (root / "CHANGELOG.md").read_text() == changelog and all((root / "changelog.d" / n).exists() for n in fragments) + check(f"refuses {name}, writes nothing", raised and untouched) + + +def dry_run_next_release(tmp: Path) -> None: + """Collect the real repo's pending fragments into a copy, under a version one minor + above the newest released one, so the check stays valid after every release.""" + real = Path(tempfile.mkdtemp(dir=tmp)) + shutil.copy(REPO / "CHANGELOG.md", real / "CHANGELOG.md") + shutil.copytree(REPO / "changelog.d", real / "changelog.d") + pending = [p.read_text().strip() for p in sorted((real / "changelog.d").glob("*.md")) if p.name != "README.md"] + if not pending: + print("skip repo dry run: no pending fragments (fresh after a release)") + return + text = (real / "CHANGELOG.md").read_text() + released = [tuple(map(int, v)) for v in re.findall(r"^## \[(\d+)\.(\d+)\.(\d+)\]", text, re.MULTILINE)] + major, minor, _ = max(released) + version = f"{major}.{minor + 1}.0" + cc.collect(real, version, "2026-10-01") + head, rest = text.split(cc.MARKER) + prefix = f"{head}{cc.MARKER}\n\n## [{version}] - 2026-10-01\n\n" + suffix = rest.lstrip("\n") + out = (real / "CHANGELOG.md").read_text() + intact = out.startswith(prefix) and out.endswith(suffix) + body = out[len(prefix) : len(out) - len(suffix)] if intact else "" + check( + f"repo: next release ({version}) holds every pending fragment verbatim", intact and body.strip() == "\n\n".join(pending) + ) + + +def main() -> int: + tmp = Path(tempfile.mkdtemp()) + try: + # The scriv failure: filename order disagrees with lexical order, and a repeated + # heading is split by another entry, so sorting, regrouping or merging all show. + frags = { + "20260930_lab-2.md": "### SaaS API\n\n- second\n", + "20260929_lab-1.md": "### Wire format\n\n- first\n continued `code`\n", + "20261001_lab-3.md": "### Wire format\n\n- third\n", + } + root = make(tmp, BASE, frags) + cc.collect(root, "1.1.0", "2026-10-01") + out = (root / "CHANGELOG.md").read_text() + want = ( + f"# Changelog\n\n## [Unreleased]\n\nPointer.\n\n{cc.MARKER}\n\n## [1.1.0] - 2026-10-01\n\n" + "### Wire format\n\n- first\n continued `code`\n\n### SaaS API\n\n- second\n\n" + "### Wire format\n\n- third\n\n" + "## [1.0.0] - 2026-03-28\n\nInitial.\n" + ) + check("section is the verbatim fragments in filename order", out == want) + left = sorted(p.name for p in (root / "changelog.d").iterdir()) + check("collected fragments deleted, README kept", left == ["README.md"]) + + # A second release lands above the first. + (root / "changelog.d" / "20261002_lab-4.md").write_text("### Later\n\n- third\n") + cc.collect(root, "1.2.0", "2026-10-02") + out2 = (root / "CHANGELOG.md").read_text() + check( + "second release sits above the first, first unchanged", + out2.index("## [1.2.0]") < out2.index("## [1.1.0]") < out2.index("## [1.0.0]") + and want.split(cc.MARKER)[1].lstrip("\n") in out2, + ) + + dry_run_next_release(tmp) + + refuses(tmp, "missing marker", BASE.replace(cc.MARKER, ""), {"a.md": "x\n"}) + refuses(tmp, "repeated marker", BASE + cc.MARKER + "\n", {"a.md": "x\n"}) + refuses(tmp, "existing version", BASE, {"a.md": "x\n"}, version="1.0.0") + refuses(tmp, "malformed version", BASE, {"a.md": "x\n"}, version="v1.1") + refuses(tmp, "no fragments", BASE, {}) + refuses(tmp, "empty fragment", BASE, {"a.md": "x\n", "b.md": " \n\n"}) + finally: + shutil.rmtree(tmp) + print(f"\n{len(FAILURES)} failure(s)") + return 1 if FAILURES else 0 + + +if __name__ == "__main__": + sys.exit(main())