feat(interop): pin untrusted-decode bounds as a cross-SDK invariant (LAB-2503)

.github/workflows/verify.yml

@@ -30,11 +30,13 @@ jobs:
           # Mutation suite first, same doctrine as the version-floor guard below:
           # prove the fail-closed guards still fail before trusting the verify.
           python3 tools/test_wire_format_reference.py
+          python3 tools/test_decode_bounds_reference.py
           python3 tools/interop-reference.py verify
           python3 tools/interop-v2-reference.py verify
           python3 tools/test_encryption_verify.py
           python3 tools/encryption-verify.py
           python3 tools/wire-format-reference.py verify
+          python3 tools/decode-bounds-reference.py verify
 
       - name: Python reference verify (optional deps — AES-GCM seal + msgpack third-encoder + lz4 C-implementation conformance)
         run: |
@@ -44,6 +46,7 @@ jobs:
           python3 tools/test_encryption_verify.py
           python3 tools/encryption-verify.py --require-seal
           python3 tools/wire-format-reference.py verify --require-extras
+          python3 tools/decode-bounds-reference.py verify --require-extras
 
       - name: JS cross-check (independent encoder + @noble/hashes + WebCrypto)
         run: |

changelog.d/20260929_lab-2503.md

@@ -0,0 +1,27 @@
+### Interop mode — untrusted-decode bounds pinned as a cross-SDK invariant (LAB-2503)
+
+- New [`spec/interop-mode.md` → Decode bounds](spec/interop-mode.md#decode-bounds):
+  readers MUST bound nesting depth (≥ 32, ≤ 1024), MUST NOT pre-allocate beyond
+  what the input can back (Σ declared slots ≤ input bytes − 1), and MUST fail
+  closed with a catchable error. Follow-up to the LAB-2487 measurements.
+- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) `1.1.0`
+  (17 reject + 3 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py),
+  which derives every vector's depth and slot tags with a structural walk, and its
+  mutation suite. The SDKs vendor earlier revisions; see the matrix's footnote 16.
+- An SDK's conformance test MUST assert that its structural guard rejects each
+  reject vector at every untrusted decode entry point (including invalidation
+  events), before materialising it; a size cap that rejects first also counts. A
+  verdict alone does not show when a reader rejected, and a direct guard call alone
+  does not show that the read path runs the guard.
+- [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits)
+  states the whole-document slot rule (per-header checks do not satisfy it; ext
+  lengths count) for the envelope bytes and the payload inside them, and the
+  Verification Flow pre-scans before it decodes.
+- Matrix: ByteStorage is ⚠️ in all three SDKs, because each decodes the envelope with
+  `cachekit-core`'s `ByteStorage::retrieve`, which has no step-2 pre-scan. The
+  decode-bounds test cells for Python and Rust are ⚠️ until their tests assert the
+  guard's rejection, and TypeScript's until its envelope entry point has a guard to
+  assert.
+- Open: the shared depth value, and any cap on the ~70× materialisation of *legal*
+  payloads, stay [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s
+  items.

sdk-feature-matrix.md

@@ -293,20 +293,20 @@ its spec:
 | Requirement | Python | Rust | TypeScript | PHP |
 | :--- | :---: | :---: | :---: | :---: |
 | Key generation (Blake2b) | ✅ Compliant | N/A auto mode¹⁴ — interop/v1 keygen ✅ merged ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); `#[cachekit]` mints interop keys ([#35](https://github.com/cachekit-io/cachekit-rs/pull/35)) | ✅ Compliant | ⚠️ Untested |
-| Wire format (ByteStorage) | ✅ Compliant¹⁵ | ✅ Canonical (`cachekit-core`) — unused for stored values¹⁵ | ✅ Compliant | ⚠️ Untested |
+| Wire format (ByteStorage) | ⚠️ Compliant¹⁵ except the envelope pre-scan: payload decodes are pre-scanned (`unpackb_bounded`), but the envelope goes through `cachekit-core`'s `ByteStorage::retrieve`, which has no [Retrieve Flow](spec/wire-format.md#retrieve-flow) step-2 pre-scan | ⚠️ Canonical (`cachekit-core`) — unused for stored values¹⁵; `ByteStorage::retrieve` decodes the envelope (typed `rmp_serde::from_slice` into `StorageEnvelope`) with no step-2 pre-scan | ⚠️ Compliant except the envelope pre-scan: payload decodes are pre-scanned, but the envelope goes through `cachekit-core`'s `ByteStorage::retrieve` (`cachekit-core-ts` `unpack`), which has no step-2 pre-scan | ⚠️ Untested |
 | Storage container (auto mode)¹⁵ | CK v3 frame (Python-internal) | Plain MessagePack (`rmp` named) — no envelope | Bare ByteStorage envelope (default) | — |
 | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested |
 | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented |
 | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented |
-| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) — fixture 1.1.0 (`ns`/`nsapi` namespace reservation) in [cachekit-py#350](https://github.com/cachekit-io/cachekit-py/pull/350), unreleased | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) — fixture 1.1.0 in [cachekit-rs#89](https://github.com/cachekit-io/cachekit-rs/pull/89), unreleased | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors — fixture 1.1.0 in [cachekit-ts#143](https://github.com/cachekit-io/cachekit-ts/pull/143), unreleased | ⚠️ Pending |
+| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) — fixture 1.1.0 (`ns`/`nsapi` namespace reservation) in [cachekit-py#350](https://github.com/cachekit-io/cachekit-py/pull/350), unreleased; `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) — fixture 1.1.0 in [cachekit-rs#89](https://github.com/cachekit-io/cachekit-rs/pull/89), unreleased; `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors — fixture 1.1.0 in [cachekit-ts#143](https://github.com/cachekit-io/cachekit-ts/pull/143), unreleased; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting the guard error each vector trips (pre-scan, or the event size cap ahead of it) — ⚠️ except the envelope entry point (`cachekit-core-ts` `unpack`), which has no guard to assert (see the Wire format row) | ⚠️ Pending |
 | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented |
 
 > [!NOTE]
 > ¹⁴ "N/A" for Rust *auto-mode* key generation means `cachekit-rs` implements no auto-mode key format: `get`/`set` take caller-supplied keys. The `#[cachekit]` macro mints **interop/v1** keys via `interop_key` — required, compile-time-validated `interop = "operation"` and `namespace` attributes, byte-identical across SDKs ([cachekit-rs#35](https://github.com/cachekit-io/cachekit-rs/pull/35) / LAB-424; keygen itself merged in [#33](https://github.com/cachekit-io/cachekit-rs/pull/33)). The legacy RFC §3.1.5 keygen (`key::generate_cache_key`, `{namespace}:{blake2b256-hex}` — matched no protocol format, and WAS live in every `#[cachekit]` expansion despite the audit's "unused" premise, a proc-macro grep miss) is deleted outright in #35; upgrading is a full cache invalidation for `#[cachekit]` users. `cachekit-core` is a protocol primitive library with no keygen.
 >
 > ¹⁵ Auto-mode **stored bytes** are SDK-internal and differ per SDK — see [wire-format.md → SDK Storage Containers](spec/wire-format.md#sdk-storage-containers-auto-mode). Python stores the ByteStorage envelope *inside* its CK v3 frame; `cachekit-rs` does not use the envelope for values at all (it uses `cachekit-core` only for encryption). Cross-SDK value compatibility is exclusively an [interop-mode](spec/interop-mode.md) property (protocol#11).
 >
-> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425).
+> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. The SDKs vendor earlier revisions of `decode-bounds.json`, all labelled `1.0.0`: cachekit-py and cachekit-ts the 13-reject / 2-accept revision, cachekit-rs a 10-reject / 2-accept one. This repo's file is `1.1.0` (17 reject, 3 accept); vectors newer than an SDK's copy run only here until that SDK re-vendors. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425).
 >
 > ¹⁷ Version cells are **floors** (`X+`), not snapshots — they stay true as new versions publish; check the registry for the current release. Python's floor is the first *installable* one: interop merged under the `v0.13.0` tag, but neither `0.12.0` nor `0.13.0` was ever published to PyPI, so `0.14.0` is the earliest PyPI release containing interop mode. Do not "correct" this to 0.13.0 from the cachekit-py changelog alone.
 

spec/interop-mode.md

@@ -39,6 +39,7 @@
 - [Encryption in Interop Mode](#encryption-in-interop-mode)
 - [SaaS Considerations](#saas-considerations)
 - [SDK Implementation Requirements](#sdk-implementation-requirements)
+  - [Decode bounds](#decode-bounds)
 - [Design Decisions](#design-decisions)
 - [Test Vectors](#test-vectors)
 
@@ -459,6 +460,79 @@ strings** (TypeScript has no UUID type): callers MUST use the lowercase hyphenat
 form, or `"550E8400-…"` from TS will silently miss the key a Python `uuid.UUID`
 argument produced.
 
+### Decode bounds
+
+Interop values are read from a backend the SDK does not control, so every decoder
+is an untrusted-input parser. A MessagePack collection header costs 1–5 bytes but
+may declare up to 2³²−1 elements, and an eager decoder pre-allocates the container
+*before* decoding its children; depth-first decoding stacks those allocations, so a
+few KB of nested headers can drive hundreds of MB of transient heap. Measured peak
+heap: 15 KB → ~400 MB in `@msgpack/msgpack` 3.1.3, and 10 KB → ~82 MB in
+`msgpack-python` 1.2.1 with `array32` headers claiming `len(input)` elements (8 bytes
+× 1024 levels × input length: it allocates every level until its nesting limit trips).
+A reader MUST therefore:
+
+1. **Bound nesting depth.** Depth is the number of collection headers on the
+   deepest path from the root. A map counts one level, like an array; str, bin, ext
+   and scalars add nothing, so `[[null]]` and `{"": [null]}` both have depth 2. The
+   bound MUST be at least 32 and MUST NOT exceed 1024.
+   (Today: TypeScript 100, Rust 100, Python 1024. A single shared value is
+   [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item;
+   until it is ratified, writers SHOULD keep values within 32 levels.) A recursive
+   native decoder can exhaust its thread stack below 1024 levels (`rmp-serde` in a
+   debug build does on a 2 MiB thread), so each SDK SHOULD test a complete document
+   at its own bound on its smallest supported stack.
+2. **Never pre-allocate beyond what the input can back.** Every declared element or
+   byte (collection elements; str, bin and ext bytes) needs at least one input byte,
+   so the declared slots summed over the whole
+   document MUST NOT exceed input bytes − 1, and a document that exceeds it MUST be
+   rejected *without* materialising it. Checking each header only against the input
+   that remains after it does not satisfy this: nested headers can each fit what
+   follows them while together declaring far more than the input holds
+   (`nested_array16_each_header_fits_sum_overclaims`). A map pair counts as two slots
+   (key + value). Exceeding the sum is sufficient to reject but does not define an
+   incomplete document: `92 dc 00 00` sums to 2 and is still truncated. A reader MUST
+   reject a structurally incomplete document as well. Every per-header term and the running sum MUST be computed in at least
+   64 bits or with checked/saturating arithmetic, and an overflow is itself a
+   rejection: two `array32` headers already exceed 2³², and a 32-bit accumulator that
+   wraps to a small value passes the budget (`array32_sum_wraps_u32`,
+   `array32_sum_wraps_u32_small_first` and `map32_half_claim_wraps_u32_mul` pin the
+   shapes). Do not assume a decoder is lazy: `rmp-serde` reads str/bin lazily but
+   serde's `Vec<T>` visitor still pre-allocates up to 1 MiB per collection from the
+   declared length. A header-only structural walk before decoding (the pre-scan in
+   `cachekit-ts`, `check_msgpack_structure` in `cachekit-py`, `check_structure` in
+   `cachekit-rs`) is sufficient.
+3. **Fail closed, catchably.** Rejection surfaces as a decode error the SDK read
+   path turns into a cache miss — never an uncaught crash or an OOM abort.
+
+These bounds are SDK-owned invariants, not library defaults: each SDK pins them
+explicitly and regression-tests them, so a decoder dependency bump cannot silently
+re-open the amplifier. A verdict cannot show that, because it does not say *when* a
+reader rejected: a stock decoder's default limits reject every reject vector today,
+and a reader with per-header checks alone rejects the incomplete ones at end of input,
+after it has pre-allocated for them. An SDK's conformance test MUST therefore assert
+that its structural guard rejects each reject vector before anything is materialised,
+by driving each reject vector through every untrusted decode entry point (value
+reads, and any other untrusted decode such as invalidation events), below the point
+where the SDK turns the error into a cache miss or drops it, and asserting an error
+that only a pre-decode check produces: the structural guard, or a size cap that entry
+point applies ahead of it. Calling the guard directly as well is fine, but on its own
+does not show that the read path runs it. A run that only asserts that a decode fails
+does not demonstrate conformance.
+[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the
+bytes every decoder MUST reject (17) and MUST accept (3); the same rules apply to
+any other untrusted MessagePack decode in an SDK (auto-mode payloads after the
+envelope is unwrapped, invalidation events).
+
+These rules do not bound the residual. A *legal*, fully backed document still
+materialises far more memory than its size in language objects: an `array32` of
+empty maps peaks at 72× its size in `msgpack-python` 1.2.1 (2 MB → 144 MB). The
+ratio applies to the decode input, which for an auto-mode payload is the
+LZ4-decompressed bytes (up to 512 MiB under
+[wire-format.md → Security Limits](wire-format.md#security-limits)), not the stored
+bytes. No normative input-size or element-count cap exists; a shared value belongs
+with the depth value on [protocol#20](https://github.com/cachekit-io/protocol/issues/20).
+
 ---
 
 ## Design Decisions
@@ -493,6 +567,10 @@ not re-litigated by accident.
 | `encryption_vectors` | 1 | Full HKDF-SHA256 → AES-256-GCM round-trip over plain-msgpack plaintext with the interop AAD (fixed nonce; decrypt-verified) |
 | `error_vectors` | 11 | Inputs that MUST be rejected (NaN, +Inf and −Inf as independent vectors, int overflow/underflow, naive datetime, bad segments incl. trailing newline, the reserved namespaces `ns` and `nsapi`). The `error` text is a maintainer note, not a normative message |
 
+[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the
+[Decode bounds](#decode-bounds); `tools/decode-bounds-reference.py verify` checks it,
+and the tool's docstring states what each CI leg proves.
+
 Inputs use a tagged-JSON convention (`{"$set": …}`, `{"$float": "2.0"}`,
 `{"$int": "…"}`, `{"$datetime": "…"}`, `{"$uuid": "…"}`, `{"$bytes": "<hex>"}`)
 documented in the file header, because JSON alone cannot express sets, bytes, floats