From bb8feb8cae667f7d07bc414c090c45fd1764c9de Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Wed, 2 Sep 2026 18:05:06 +1000 Subject: [PATCH 01/10] feat(interop): pin untrusted-decode bounds as a cross-SDK invariant (LAB-2503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Readers MUST bound nesting depth (32..=1024) and MUST NOT pre-allocate beyond what the input can back; test-vectors/decode-bounds.json pins 10 reject + 2 accept vectors, generated/verified by tools/decode-bounds-reference.py (stdlib; optional msgpack-python leg). Motivation: LAB-2487 measured nested-header amplification in eager decoders. The 82 MB ceiling previously reported for msgpack-python was an artifact of the array16(10000) probe — array32 headers claiming len(input) reach ~8192x input (10 KB -> 67 MB). --- .github/workflows/verify.yml | 2 + CHANGELOG.md | 22 ++++ sdk-feature-matrix.md | 2 +- spec/interop-mode.md | 46 +++++++ spec/wire-format.md | 6 +- test-vectors/decode-bounds.json | 210 +++++++++++++++++++++++++++++++ tools/decode-bounds-reference.py | 193 ++++++++++++++++++++++++++++ 7 files changed, 479 insertions(+), 2 deletions(-) create mode 100644 test-vectors/decode-bounds.json create mode 100644 tools/decode-bounds-reference.py diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index a267e98..578b620 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -34,6 +34,7 @@ jobs: python3 tools/interop-v2-reference.py verify python3 tools/encryption-verify.py python3 tools/wire-format-reference.py verify + python3 tools/decode-bounds-reference.py verify - name: Python reference verify (optional deps — AES-GCM seal + msgpack third-encoder + lz4 C-implementation conformance) run: | @@ -42,6 +43,7 @@ jobs: python3 tools/interop-v2-reference.py verify python3 tools/encryption-verify.py --require-seal python3 tools/wire-format-reference.py verify --require-extras + python3 tools/decode-bounds-reference.py verify - name: JS cross-check (independent encoder + @noble/hashes + WebCrypto) run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 7954064..a1f3511 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,28 @@ All notable changes to the CacheKit Protocol Specification. ## [Unreleased] +### Interop mode — untrusted-decode bounds pinned as a cross-SDK invariant (LAB-2503) + +- New [`spec/interop-mode.md` → Decode bounds](spec/interop-mode.md#decode-bounds): + readers MUST bound nesting depth (≥ 32, ≤ 1024), MUST NOT pre-allocate beyond + what the input can back (Σ declared slots ≤ input bytes − 1; structurally + incomplete documents are rejected without being materialised), and MUST fail + closed with a catchable error. Motivated by the LAB-2487 measurements: nested + collection headers amplify a few KB of input into hundreds of MB of transient + heap in eager decoders (15 KB → ~400 MB in `@msgpack/msgpack`; 10 KB → 67 MB + in `msgpack-python` with `array32` headers claiming `len(input)` — the + "82 MB hard ceiling" previously reported for Python was an artifact of the + `array16(10000)` probe, not a property of the decoder). +- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json): + 10 reject vectors + 2 accept vectors, generated and verified by + [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py) + (stdlib; the optional-deps CI leg also proves `msgpack-python` conforms). + Vendored and CI-executed by cachekit-py and cachekit-rs. +- [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) + cross-references the rules for the payload inside the envelope. +- The single shared depth value stays [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s + open item; the spec pins the floor/ceiling every SDK already satisfies. + ### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) - LZ4 compressed bytes are **not canonical** across conforming block encoders. diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index a1f64e9..9d8bf09 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -285,7 +285,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) + decode-bounds (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) + decode-bounds (LAB-2503) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)), `decode-bounds.json` not yet vendored | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 6948dc2..d0ada36 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -35,6 +35,7 @@ - [Encryption in Interop Mode](#encryption-in-interop-mode) - [SaaS Considerations](#saas-considerations) - [SDK Implementation Requirements](#sdk-implementation-requirements) + - [Decode bounds](#decode-bounds) - [Design Decisions](#design-decisions) - [Test Vectors](#test-vectors) @@ -438,6 +439,43 @@ strings** (TypeScript has no UUID type): callers MUST use the lowercase hyphenat form, or `"550E8400-…"` from TS will silently miss the key a Python `uuid.UUID` argument produced. +### Decode bounds + +Interop values are read from a backend the SDK does not control, so every decoder +is an untrusted-input parser. A MessagePack collection header costs 1–5 bytes but +may declare up to 2³²−1 elements, and an eager decoder pre-allocates the container +*before* decoding its children; depth-first decoding stacks those allocations, so a +few KB of nested headers can drive hundreds of MB of transient heap (measured +15 KB → ~400 MB in `@msgpack/msgpack` 3.1.3; 10 KB → 67 MB in `msgpack-python` +1.2.1 with `array32` headers claiming `len(input)` elements). A reader MUST +therefore: + +1. **Bound nesting depth.** The bound MUST be at least 32 and MUST NOT exceed 1024. + (Today: TypeScript 100, Rust 100 — `rmp-serde`'s 1024 default overflows a + 2 MiB thread stack in debug builds, an uncatchable abort — Python 1024, fixed + by `msgpack-python`'s iterative C unpacker and not configurable. A single shared value is + [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item; + until it is ratified, writers SHOULD keep values within 32 levels.) +2. **Never pre-allocate beyond what the input can back.** Every declared element or + byte needs at least one input byte, so a structurally incomplete document + (Σ declared slots > input bytes − 1) MUST be rejected *without* materialising it. + Slice-based decoders that size containers lazily satisfy this inherently + (`rmp-serde`); decoders that pre-allocate from headers MUST validate first — a + header-only structural walk (`Unpacker.skip()` in `msgpack-python`, the pre-scan + in `cachekit-ts`) is sufficient. +3. **Fail closed, catchably.** Rejection surfaces as a decode error the SDK read + path turns into a cache miss — never an uncaught crash or an OOM abort. + +These bounds are SDK-owned invariants, not library defaults: each SDK pins them +explicitly and regression-tests them, so a decoder dependency bump cannot silently +re-open the amplifier. +[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the +bytes every decoder MUST reject (10) and MUST accept (2); the same rules apply to +any other untrusted MessagePack decode in an SDK (auto-mode payloads after the +envelope is unwrapped, invalidation events). The 40× residual — a *legal* payload +still materialises far more than its byte size in language objects — is bounded by +each SDK's input-size cap, not by these rules. + --- ## Design Decisions @@ -472,6 +510,14 @@ not re-litigated by accident. | `encryption_vectors` | 1 | Full HKDF-SHA256 → AES-256-GCM round-trip over plain-msgpack plaintext with the interop AAD (fixed nonce; decrypt-verified) | | `error_vectors` | 9 | Inputs that MUST be rejected (NaN, +Inf and −Inf as independent vectors, int overflow/underflow, naive datetime, bad segments incl. trailing newline). The `error` text is a maintainer note, not a normative message | +[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) (see +[Decode bounds](#decode-bounds)) adds 10 `reject_vectors` (nested-header bombs, +over-claiming `array32`/`map32`/`bin32`/`str32` headers, a truncated array) and 2 +`accept_vectors` (32-deep nesting, a fully backed `array16`) that every SDK's +untrusted decoder MUST honour; `tools/decode-bounds-reference.py verify` checks the +file against its recipes and, when `msgpack-python` is installed, against the real +decoder. + Inputs use a tagged-JSON convention (`{"$set": …}`, `{"$float": "2.0"}`, `{"$int": "…"}`, `{"$datetime": "…"}`, `{"$uuid": "…"}`, `{"$bytes": ""}`) documented in the file header, because JSON alone cannot express sets, bytes, floats diff --git a/spec/wire-format.md b/spec/wire-format.md index 74e5e89..cebffe1 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -377,7 +377,11 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes(); > length header against the remaining input bytes **before** allocating for it — > a 5-byte `bin32` header can otherwise declare a 4 GiB allocation from a > ~30-byte envelope. (Slice-based decoders such as `rmp-serde` satisfy this -> inherently; readers that pre-allocate from length fields must check.) +> inherently; readers that pre-allocate from length fields must check.) The +> payload *inside* the envelope is untrusted MessagePack too — decode it under +> the depth and allocation rules in +> [interop-mode.md → Decode bounds](interop-mode.md#decode-bounds), pinned by +> `test-vectors/decode-bounds.json`. | Limit | Value | Purpose | | :--- | ---: | :--- | diff --git a/test-vectors/decode-bounds.json b/test-vectors/decode-bounds.json new file mode 100644 index 0000000..91ab717 --- /dev/null +++ b/test-vectors/decode-bounds.json @@ -0,0 +1,210 @@ +{ + "version": "1.0.0", + "spec": "spec/interop-mode.md#decode-bounds", + "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", + "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, auto-mode payloads after the ByteStorage envelope is unwrapped, invalidation events. The bytes are plain MessagePack with no envelope.", + "rules": { + "depth": "Readers MUST bound nesting depth. The bound MUST be >= 32 and MUST be <= 1024; every reject vector tagged 'depth' nests deeper than 1024.", + "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining input can back (each element or byte needs >= 1 input byte), and MUST reject a structurally incomplete document. Every reject vector tagged 'overclaim' has declared_slots > input_len - 1 (the root header is the only byte that is not an element).", + "failure_mode": "Rejection MUST surface as a catchable decode error that the SDK read path turns into a cache miss (fail-closed), never an uncaught crash or an OOM abort." + }, + "field_notes": { + "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", + "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", + "declared_slots": "sum of every header's declared element/byte count (a nested header counts as one element of its parent)", + "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message" + }, + "reject_vectors": [ + { + "name": "nested_array16_depth_2048", + "description": "2048 nested array16 headers each claiming 10 000 elements, 0 backing bytes. The LAB-2487 amplifier shape: an eager decoder pre-allocates 10 000 slots per level before hitting EOF.", + "construction": { + "repeat_hex": "dc2710", + "count": 2048, + "suffix_hex": "" + }, + "input_hex": "dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710", + "input_len": 6144, + "nesting_depth": 2048, + "declared_slots": 20480000, + "reject_reasons": [ + "depth", + "overclaim" + ] + }, + { + "name": "nested_array32_input_len_depth_1100", + "description": "1100 nested array32 headers each claiming exactly len(input)=5500 elements. Defeats a per-collection cap of len(input): peak pre-allocation is depth x len(input) x slot size.", + "construction": { + "repeat_hex": "dd0000157c", + "count": 1100, + "suffix_hex": "" + }, + "input_hex": "dd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157cdd0000157c", + "input_len": 5500, + "nesting_depth": 1100, + "declared_slots": 6050000, + "reject_reasons": [ + "depth", + "overclaim" + ] + }, + { + "name": "nested_map16_depth_2048", + "description": "Map twin of nested_array16_depth_2048 (map pre-allocation is typically larger per slot).", + "construction": { + "repeat_hex": "de2710", + "count": 2048, + "suffix_hex": "" + }, + "input_hex": "de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710", + "input_len": 6144, + "nesting_depth": 2048, + "declared_slots": 20480000, + "reject_reasons": [ + "depth", + "overclaim" + ] + }, + { + "name": "nested_fixarray_depth_2048_complete", + "description": "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", + "construction": { + "repeat_hex": "91", + "count": 2048, + "suffix_hex": "c0" + }, + "input_hex": "9191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191c0", + "input_len": 2049, + "nesting_depth": 2048, + "declared_slots": 2048, + "reject_reasons": [ + "depth" + ] + }, + { + "name": "array16_overclaim_shallow", + "description": "One array16 header claiming 10 000 elements with 3 backing bytes.", + "construction": { + "repeat_hex": "dc2710", + "count": 1, + "suffix_hex": "010203" + }, + "input_hex": "dc2710010203", + "input_len": 6, + "nesting_depth": 1, + "declared_slots": 10000, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "array32_max_claim_alone", + "description": "A lone 5-byte array32 header claiming 2^32-1 elements.", + "construction": { + "repeat_hex": "ddffffffff", + "count": 1, + "suffix_hex": "" + }, + "input_hex": "ddffffffff", + "input_len": 5, + "nesting_depth": 1, + "declared_slots": 4294967295, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "map32_max_claim_alone", + "description": "A lone 5-byte map32 header claiming 2^32-1 pairs.", + "construction": { + "repeat_hex": "dfffffffff", + "count": 1, + "suffix_hex": "" + }, + "input_hex": "dfffffffff", + "input_len": 5, + "nesting_depth": 1, + "declared_slots": 4294967295, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "bin32_overclaim", + "description": "bin32 header claiming 2^32-1 bytes with 1 backing byte (a 6-byte document declaring a 4 GiB buffer).", + "construction": { + "repeat_hex": "c6ffffffff", + "count": 1, + "suffix_hex": "41" + }, + "input_hex": "c6ffffffff41", + "input_len": 6, + "nesting_depth": 0, + "declared_slots": 4294967295, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "str32_overclaim", + "description": "str32 twin of bin32_overclaim.", + "construction": { + "repeat_hex": "dbffffffff", + "count": 1, + "suffix_hex": "41" + }, + "input_hex": "dbffffffff41", + "input_len": 6, + "nesting_depth": 0, + "declared_slots": 4294967295, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "fixarray_short_by_one", + "description": "fixarray claiming 5 elements with 4 present: the minimal truncated document.", + "construction": { + "repeat_hex": "95", + "count": 1, + "suffix_hex": "c0c0c0c0" + }, + "input_hex": "95c0c0c0c0", + "input_len": 5, + "nesting_depth": 1, + "declared_slots": 5, + "reject_reasons": [ + "overclaim" + ] + } + ], + "accept_vectors": [ + { + "name": "nested_fixarray_depth_32", + "description": "[[...[null]...]] nested 32 deep, complete. A conforming reader MUST accept it: the depth bound may not be tighter than 32.", + "construction": { + "repeat_hex": "91", + "count": 32, + "suffix_hex": "c0" + }, + "input_hex": "9191919191919191919191919191919191919191919191919191919191919191c0", + "input_len": 33, + "nesting_depth": 32, + "declared_slots": 32 + }, + { + "name": "array16_256_backed_nils", + "description": "array16 header claiming 256 elements with all 256 present. A *16 header that is fully backed by input is legitimate; the allocation rule is about backing, not header width.", + "construction": { + "repeat_hex": "dc0100", + "count": 1, + "suffix_hex": "c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0" + }, + "input_hex": "dc0100c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0", + "input_len": 259, + "nesting_depth": 1, + "declared_slots": 256 + } + ] +} diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py new file mode 100644 index 0000000..2b8b369 --- /dev/null +++ b/tools/decode-bounds-reference.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python3 +"""Reference tool for test-vectors/decode-bounds.json (untrusted-decode bounds). + +Why these vectors exist: a MessagePack collection header costs 1-5 bytes but can +declare up to 2^32-1 elements, and an eager decoder pre-allocates the container +BEFORE decoding the children. Depth-first decoding stacks those allocations, so +a few KB of nested headers can drive hundreds of MB of transient heap (measured +15 KB -> ~400 MB in @msgpack/msgpack 3.1.3; 10 KB -> 67 MB in msgpack-python +1.2.1 with array32 headers claiming len(input) elements). Normative rules live +in spec/interop-mode.md "Decode bounds"; this file pins the bytes every SDK's +decoder MUST reject (and two it MUST accept, so the bound cannot over-tighten). + +Usage: + verify (default) stdlib-only. Regenerates every vector from its + `construction` recipe and byte-compares to `input_hex`; checks the + declared depth / slot arithmetic. When `msgpack` (msgpack-python) + is importable, additionally proves the real decoder rejects every + reject vector and accepts every accept vector. + generate Rewrites the vector file from the recipes below. +""" + +from __future__ import annotations + +import json +from pathlib import Path +import sys + +ROOT = Path(__file__).resolve().parents[1] +VECTORS = ROOT / "test-vectors" / "decode-bounds.json" + +# Every SDK's current depth bound is <= MAX_DEPTH_CEILING (ts 100, rs 100, py +# 1024), so a reject vector nested deeper than this is rejected by all of them; +# every SDK accepts at least MIN_DEPTH_FLOOR levels. +MAX_DEPTH_CEILING = 1024 +MIN_DEPTH_FLOOR = 32 + + +def u16(n: int) -> str: + return n.to_bytes(2, "big").hex() + + +def u32(n: int) -> str: + return n.to_bytes(4, "big").hex() + + +def recipe(name: str, description: str, repeat_hex: str, count: int, suffix_hex: str = "", *, + depth: int, slots: int, reasons: list[str]) -> dict: + data = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex) + return { + "name": name, + "description": description, + "construction": {"repeat_hex": repeat_hex, "count": count, "suffix_hex": suffix_hex}, + "input_hex": data.hex(), + "input_len": len(data), + "nesting_depth": depth, + "declared_slots": slots, + "reject_reasons": reasons, + } + + +def build() -> dict: + reject = [ + recipe("nested_array16_depth_2048", + "2048 nested array16 headers each claiming 10 000 elements, 0 backing bytes. The LAB-2487 " + "amplifier shape: an eager decoder pre-allocates 10 000 slots per level before hitting EOF.", + "dc" + u16(10000), 2048, depth=2048, slots=2048 * 10000, reasons=["depth", "overclaim"]), + recipe("nested_array32_input_len_depth_1100", + "1100 nested array32 headers each claiming exactly len(input)=5500 elements. Defeats a " + "per-collection cap of len(input): peak pre-allocation is depth x len(input) x slot size.", + "dd" + u32(5500), 1100, depth=1100, slots=1100 * 5500, reasons=["depth", "overclaim"]), + recipe("nested_map16_depth_2048", + "Map twin of nested_array16_depth_2048 (map pre-allocation is typically larger per slot).", + "de" + u16(10000), 2048, depth=2048, slots=2048 * 10000, reasons=["depth", "overclaim"]), + recipe("nested_fixarray_depth_2048_complete", + "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, " + "so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", + "91", 2048, "c0", depth=2048, slots=2048, reasons=["depth"]), + recipe("array16_overclaim_shallow", + "One array16 header claiming 10 000 elements with 3 backing bytes.", + "dc" + u16(10000), 1, "010203", depth=1, slots=10000, reasons=["overclaim"]), + recipe("array32_max_claim_alone", + "A lone 5-byte array32 header claiming 2^32-1 elements.", + "dd" + u32(0xFFFFFFFF), 1, depth=1, slots=0xFFFFFFFF, reasons=["overclaim"]), + recipe("map32_max_claim_alone", + "A lone 5-byte map32 header claiming 2^32-1 pairs.", + "df" + u32(0xFFFFFFFF), 1, depth=1, slots=0xFFFFFFFF, reasons=["overclaim"]), + recipe("bin32_overclaim", + "bin32 header claiming 2^32-1 bytes with 1 backing byte (a 6-byte document declaring a 4 GiB buffer).", + "c6" + u32(0xFFFFFFFF), 1, "41", depth=0, slots=0xFFFFFFFF, reasons=["overclaim"]), + recipe("str32_overclaim", + "str32 twin of bin32_overclaim.", + "db" + u32(0xFFFFFFFF), 1, "41", depth=0, slots=0xFFFFFFFF, reasons=["overclaim"]), + recipe("fixarray_short_by_one", + "fixarray claiming 5 elements with 4 present: the minimal truncated document.", + "95", 1, "c0c0c0c0", depth=1, slots=5, reasons=["overclaim"]), + ] + accept = [ + recipe("nested_fixarray_depth_32", + "[[...[null]...]] nested 32 deep, complete. A conforming reader MUST accept it: the depth " + "bound may not be tighter than 32.", + "91", MIN_DEPTH_FLOOR, "c0", depth=MIN_DEPTH_FLOOR, slots=MIN_DEPTH_FLOOR, reasons=[]), + recipe("array16_256_backed_nils", + "array16 header claiming 256 elements with all 256 present. A *16 header that is fully " + "backed by input is legitimate; the allocation rule is about backing, not header width.", + "dc" + u16(256), 1, "c0" * 256, depth=1, slots=256, reasons=[]), + ] + for v in accept: + del v["reject_reasons"] + return { + "version": "1.0.0", + "spec": "spec/interop-mode.md#decode-bounds", + "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", + "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, auto-mode payloads after " + "the ByteStorage envelope is unwrapped, invalidation events. The bytes are plain MessagePack " + "with no envelope.", + "rules": { + "depth": f"Readers MUST bound nesting depth. The bound MUST be >= {MIN_DEPTH_FLOOR} and MUST be " + f"<= {MAX_DEPTH_CEILING}; every reject vector tagged 'depth' nests deeper than " + f"{MAX_DEPTH_CEILING}.", + "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining " + "input can back (each element or byte needs >= 1 input byte), and MUST reject a " + "structurally incomplete document. Every reject vector tagged 'overclaim' has " + "declared_slots > input_len - 1 (the root header is the only byte that is not an element).", + "failure_mode": "Rejection MUST surface as a catchable decode error that the SDK read path turns " + "into a cache miss (fail-closed), never an uncaught crash or an OOM abort.", + }, + "field_notes": { + "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", + "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", + "declared_slots": "sum of every header's declared element/byte count (a nested header counts as one element of its parent)", + "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message", + }, + "reject_vectors": reject, + "accept_vectors": accept, + } + + +def check(condition: bool, name: str, detail: str) -> None: + """Fail closed even under ``python -O`` (asserts would be stripped).""" + if not condition: + raise ValueError(f"{name}: {detail}") + + +def verify(document: dict) -> tuple[int, str]: + fresh = build() + check(document == fresh, "document", "vector file differs from the recipes; run `generate`") + for v in document["reject_vectors"] + document["accept_vectors"]: + c = v["construction"] + data = bytes.fromhex(c["repeat_hex"]) * c["count"] + bytes.fromhex(c["suffix_hex"]) + check(data.hex() == v["input_hex"], v["name"], "input_hex does not match construction") + check(len(data) == v["input_len"], v["name"], "input_len mismatch") + reasons = v.get("reject_reasons", []) + check(("depth" in reasons) == (v["nesting_depth"] > MAX_DEPTH_CEILING), v["name"], "depth tag mismatch") + # Slot budget: every declared element (including a nested header) costs >= 1 input + # byte; only the root header is not itself an element. So sum(declared) <= len - 1. + check(("overclaim" in reasons) == (v["declared_slots"] > v["input_len"] - 1), v["name"], "overclaim tag mismatch") + if not reasons: + check(v["nesting_depth"] <= MIN_DEPTH_FLOOR, v["name"], "accept vector deeper than the floor") + + try: + import msgpack # type: ignore[import-not-found] + except ImportError: + return len(document["reject_vectors"]) + len(document["accept_vectors"]), "stdlib only (msgpack absent)" + + for v in document["reject_vectors"]: + data = bytes.fromhex(v["input_hex"]) + try: + msgpack.unpackb(data) + except Exception: # noqa: BLE001 — any decode error is a conforming rejection + continue + raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") + for v in document["accept_vectors"]: + msgpack.unpackb(bytes.fromhex(v["input_hex"])) + return len(document["reject_vectors"]) + len(document["accept_vectors"]), f"msgpack-python {msgpack.version} conformance" + + +def main() -> None: + mode = sys.argv[1] if len(sys.argv) > 1 else "verify" + if mode == "generate": + VECTORS.write_text(json.dumps(build(), indent=2) + "\n", encoding="utf-8") + print(f"wrote {VECTORS.relative_to(ROOT)}") + return + if mode != "verify": + sys.exit(f"usage: {sys.argv[0]} [verify|generate]") + try: + count, leg = verify(json.loads(VECTORS.read_text(encoding="utf-8"))) + except ValueError as e: + sys.exit(f"decode-bounds verify FAILED: {e}") + print(f"decode-bounds: {count} vectors OK ({leg})") + + +if __name__ == "__main__": + main() From a4bda44fd894455dc9d124fb7fd926b5b4ab846a Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Wed, 2 Sep 2026 18:52:30 +1000 Subject: [PATCH 02/10] docs(interop): apply LAB-2503 panel findings to decode-bounds spec and tool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - rmp-serde is not allocation-free: serde's Vec visitor pre-allocates up to 1 MiB per collection from the declared length; the spec no longer claims slice-based decoders satisfy the allocation rule inherently. - nested_array16/map16 bombs now claim 2000 elements (< input_len) so a per-collection cap of len(input) does not reject them — the vectors must discriminate for msgpack-python too. - Trim repeated measurement prose; verify() drops checks already implied by the recipe equality; 'conformance' wording narrowed to reject/accept. --- CHANGELOG.md | 20 +++++------------ spec/interop-mode.md | 18 +++++++-------- test-vectors/decode-bounds.json | 14 ++++++------ tools/decode-bounds-reference.py | 38 ++++++++++++++------------------ 4 files changed, 38 insertions(+), 52 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a1f3511..76de96d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,23 +8,15 @@ All notable changes to the CacheKit Protocol Specification. - New [`spec/interop-mode.md` → Decode bounds](spec/interop-mode.md#decode-bounds): readers MUST bound nesting depth (≥ 32, ≤ 1024), MUST NOT pre-allocate beyond - what the input can back (Σ declared slots ≤ input bytes − 1; structurally - incomplete documents are rejected without being materialised), and MUST fail - closed with a catchable error. Motivated by the LAB-2487 measurements: nested - collection headers amplify a few KB of input into hundreds of MB of transient - heap in eager decoders (15 KB → ~400 MB in `@msgpack/msgpack`; 10 KB → 67 MB - in `msgpack-python` with `array32` headers claiming `len(input)` — the - "82 MB hard ceiling" previously reported for Python was an artifact of the - `array16(10000)` probe, not a property of the decoder). -- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json): - 10 reject vectors + 2 accept vectors, generated and verified by - [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py) - (stdlib; the optional-deps CI leg also proves `msgpack-python` conforms). - Vendored and CI-executed by cachekit-py and cachekit-rs. + what the input can back (Σ declared slots ≤ input bytes − 1), and MUST fail + closed with a catchable error. Follow-up to the LAB-2487 measurements. +- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) + (10 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py); + vendored and CI-executed by cachekit-py and cachekit-rs. - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) cross-references the rules for the payload inside the envelope. - The single shared depth value stays [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s - open item; the spec pins the floor/ceiling every SDK already satisfies. + open item. ### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) diff --git a/spec/interop-mode.md b/spec/interop-mode.md index d0ada36..2fcc7ca 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -451,18 +451,17 @@ few KB of nested headers can drive hundreds of MB of transient heap (measured therefore: 1. **Bound nesting depth.** The bound MUST be at least 32 and MUST NOT exceed 1024. - (Today: TypeScript 100, Rust 100 — `rmp-serde`'s 1024 default overflows a - 2 MiB thread stack in debug builds, an uncatchable abort — Python 1024, fixed - by `msgpack-python`'s iterative C unpacker and not configurable. A single shared value is + (Today: TypeScript 100, Rust 100, Python 1024. A single shared value is [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item; until it is ratified, writers SHOULD keep values within 32 levels.) 2. **Never pre-allocate beyond what the input can back.** Every declared element or byte needs at least one input byte, so a structurally incomplete document (Σ declared slots > input bytes − 1) MUST be rejected *without* materialising it. - Slice-based decoders that size containers lazily satisfy this inherently - (`rmp-serde`); decoders that pre-allocate from headers MUST validate first — a - header-only structural walk (`Unpacker.skip()` in `msgpack-python`, the pre-scan - in `cachekit-ts`) is sufficient. + Do not assume a decoder is lazy: `rmp-serde` reads str/bin lazily but serde's + `Vec` visitor still pre-allocates up to 1 MiB per collection from the + declared length. A header-only structural walk before decoding (the pre-scan in + `cachekit-ts`, `Unpacker.skip()` in `cachekit-py`, `check_structure` in + `cachekit-rs`) is sufficient. 3. **Fail closed, catchably.** Rejection surfaces as a decode error the SDK read path turns into a cache miss — never an uncaught crash or an OOM abort. @@ -515,8 +514,9 @@ not re-litigated by accident. over-claiming `array32`/`map32`/`bin32`/`str32` headers, a truncated array) and 2 `accept_vectors` (32-deep nesting, a fully backed `array16`) that every SDK's untrusted decoder MUST honour; `tools/decode-bounds-reference.py verify` checks the -file against its recipes and, when `msgpack-python` is installed, against the real -decoder. +file against its recipes and, when `msgpack-python` is installed, that the real +decoder rejects/accepts each vector (rejection only — the allocation rule is each +SDK's own guard). Inputs use a tagged-JSON convention (`{"$set": …}`, `{"$float": "2.0"}`, `{"$int": "…"}`, `{"$datetime": "…"}`, `{"$uuid": "…"}`, `{"$bytes": ""}`) diff --git a/test-vectors/decode-bounds.json b/test-vectors/decode-bounds.json index 91ab717..6a58520 100644 --- a/test-vectors/decode-bounds.json +++ b/test-vectors/decode-bounds.json @@ -17,16 +17,16 @@ "reject_vectors": [ { "name": "nested_array16_depth_2048", - "description": "2048 nested array16 headers each claiming 10 000 elements, 0 backing bytes. The LAB-2487 amplifier shape: an eager decoder pre-allocates 10 000 slots per level before hitting EOF.", + "description": "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The LAB-2487 amplifier shape: an eager decoder pre-allocates 2000 slots per level before hitting EOF. 2000 < input_len, so a per-collection cap of len(input) does NOT reject it.", "construction": { - "repeat_hex": "dc2710", + "repeat_hex": "dc07d0", "count": 2048, "suffix_hex": "" }, - "input_hex": "dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710dc2710", + "input_hex": "dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0", "input_len": 6144, "nesting_depth": 2048, - "declared_slots": 20480000, + "declared_slots": 4096000, "reject_reasons": [ "depth", "overclaim" @@ -53,14 +53,14 @@ "name": "nested_map16_depth_2048", "description": "Map twin of nested_array16_depth_2048 (map pre-allocation is typically larger per slot).", "construction": { - "repeat_hex": "de2710", + "repeat_hex": "de07d0", "count": 2048, "suffix_hex": "" }, - "input_hex": "de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710de2710", + "input_hex": "de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0de07d0", "input_len": 6144, "nesting_depth": 2048, - "declared_slots": 20480000, + "declared_slots": 8192000, "reject_reasons": [ "depth", "overclaim" diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index 2b8b369..d729416 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -1,21 +1,16 @@ #!/usr/bin/env python3 """Reference tool for test-vectors/decode-bounds.json (untrusted-decode bounds). -Why these vectors exist: a MessagePack collection header costs 1-5 bytes but can -declare up to 2^32-1 elements, and an eager decoder pre-allocates the container -BEFORE decoding the children. Depth-first decoding stacks those allocations, so -a few KB of nested headers can drive hundreds of MB of transient heap (measured -15 KB -> ~400 MB in @msgpack/msgpack 3.1.3; 10 KB -> 67 MB in msgpack-python -1.2.1 with array32 headers claiming len(input) elements). Normative rules live -in spec/interop-mode.md "Decode bounds"; this file pins the bytes every SDK's -decoder MUST reject (and two it MUST accept, so the bound cannot over-tighten). +Normative rules and the measurements behind them: spec/interop-mode.md → Decode +bounds. This file pins the bytes every SDK's decoder MUST reject (10) and MUST +accept (2, so the bound cannot over-tighten). Usage: - verify (default) stdlib-only. Regenerates every vector from its - `construction` recipe and byte-compares to `input_hex`; checks the - declared depth / slot arithmetic. When `msgpack` (msgpack-python) - is importable, additionally proves the real decoder rejects every - reject vector and accepts every accept vector. + verify (default) stdlib-only. Checks the file equals the recipes below and + that each vector's depth/slot tags match its arithmetic. When + `msgpack` (msgpack-python) is importable, additionally checks the + real decoder rejects every reject vector and accepts every accept + vector — rejection only; the allocation rule is each SDK's guard. generate Rewrites the vector file from the recipes below. """ @@ -61,16 +56,17 @@ def recipe(name: str, description: str, repeat_hex: str, count: int, suffix_hex: def build() -> dict: reject = [ recipe("nested_array16_depth_2048", - "2048 nested array16 headers each claiming 10 000 elements, 0 backing bytes. The LAB-2487 " - "amplifier shape: an eager decoder pre-allocates 10 000 slots per level before hitting EOF.", - "dc" + u16(10000), 2048, depth=2048, slots=2048 * 10000, reasons=["depth", "overclaim"]), + "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The LAB-2487 " + "amplifier shape: an eager decoder pre-allocates 2000 slots per level before hitting EOF. " + "2000 < input_len, so a per-collection cap of len(input) does NOT reject it.", + "dc" + u16(2000), 2048, depth=2048, slots=2048 * 2000, reasons=["depth", "overclaim"]), recipe("nested_array32_input_len_depth_1100", "1100 nested array32 headers each claiming exactly len(input)=5500 elements. Defeats a " "per-collection cap of len(input): peak pre-allocation is depth x len(input) x slot size.", "dd" + u32(5500), 1100, depth=1100, slots=1100 * 5500, reasons=["depth", "overclaim"]), recipe("nested_map16_depth_2048", "Map twin of nested_array16_depth_2048 (map pre-allocation is typically larger per slot).", - "de" + u16(10000), 2048, depth=2048, slots=2048 * 10000, reasons=["depth", "overclaim"]), + "de" + u16(2000), 2048, depth=2048, slots=2048 * 2 * 2000, reasons=["depth", "overclaim"]), recipe("nested_fixarray_depth_2048_complete", "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, " "so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", @@ -144,11 +140,9 @@ def check(condition: bool, name: str, detail: str) -> None: def verify(document: dict) -> tuple[int, str]: fresh = build() check(document == fresh, "document", "vector file differs from the recipes; run `generate`") + # input_hex / input_len are derived from `construction` by recipe(), so the equality + # above already proves them; what still needs checking is the hand-entered tags. for v in document["reject_vectors"] + document["accept_vectors"]: - c = v["construction"] - data = bytes.fromhex(c["repeat_hex"]) * c["count"] + bytes.fromhex(c["suffix_hex"]) - check(data.hex() == v["input_hex"], v["name"], "input_hex does not match construction") - check(len(data) == v["input_len"], v["name"], "input_len mismatch") reasons = v.get("reject_reasons", []) check(("depth" in reasons) == (v["nesting_depth"] > MAX_DEPTH_CEILING), v["name"], "depth tag mismatch") # Slot budget: every declared element (including a nested header) costs >= 1 input @@ -171,7 +165,7 @@ def verify(document: dict) -> tuple[int, str]: raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") for v in document["accept_vectors"]: msgpack.unpackb(bytes.fromhex(v["input_hex"])) - return len(document["reject_vectors"]) + len(document["accept_vectors"]), f"msgpack-python {msgpack.version} conformance" + return len(document["reject_vectors"]) + len(document["accept_vectors"]), f"msgpack-python {msgpack.version} rejects/accepts as required" def main() -> None: From 277db510a8c7981582b681fb95276ba8da75e429 Mon Sep 17 00:00:00 2001 From: Winston Date: Thu, 3 Sep 2026 09:27:57 +1000 Subject: [PATCH 03/10] fix(tools): address Kody review on decode-bounds-reference (LAB-2503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Narrow the msgpack-python rejection check from `except Exception` to `except ValueError`. Every msgpack unpack error subclasses ValueError (StackError, FormatError, ExtraData, max_*_len, incomplete input) — verified per vector under msgpack 1.0.3, 1.2.1 C extension and 1.2.1 pure-Python fallback. The broad clause was also wrong on the merits: it would have counted a MemoryError as a "conforming rejection", which is the exact failure the spec's failure_mode rule forbids. Now anything that is not a ValueError propagates and fails the run. Report lines move from print() to logging.info on a stdout handler with a message-only format, matching interop-v2-reference.py and file-backend-reference.py; stdout bytes are unchanged. Kody-Resolved: tools/decode-bounds-reference.py:163:specific exception handling Kody-Resolved: tools/decode-bounds-reference.py:175:print statements with logging --- tools/decode-bounds-reference.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index d729416..856304a 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -17,6 +17,7 @@ from __future__ import annotations import json +import logging from pathlib import Path import sys @@ -160,7 +161,10 @@ def verify(document: dict) -> tuple[int, str]: data = bytes.fromhex(v["input_hex"]) try: msgpack.unpackb(data) - except Exception: # noqa: BLE001 — any decode error is a conforming rejection + # Every msgpack-python unpack error subclasses ValueError (StackError, FormatError, + # ExtraData, max_*_len, incomplete input). Anything else — MemoryError, RecursionError, + # a crash — is the failure_mode rule being violated, so it must propagate, not count. + except ValueError: continue raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") for v in document["accept_vectors"]: @@ -172,7 +176,7 @@ def main() -> None: mode = sys.argv[1] if len(sys.argv) > 1 else "verify" if mode == "generate": VECTORS.write_text(json.dumps(build(), indent=2) + "\n", encoding="utf-8") - print(f"wrote {VECTORS.relative_to(ROOT)}") + logging.info("wrote %s", VECTORS.relative_to(ROOT)) return if mode != "verify": sys.exit(f"usage: {sys.argv[0]} [verify|generate]") @@ -180,8 +184,9 @@ def main() -> None: count, leg = verify(json.loads(VECTORS.read_text(encoding="utf-8"))) except ValueError as e: sys.exit(f"decode-bounds verify FAILED: {e}") - print(f"decode-bounds: {count} vectors OK ({leg})") + logging.info("decode-bounds: %d vectors OK (%s)", count, leg) if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout) main() From 2d56cce231e193141f09df9316f9afac17a1538e Mon Sep 17 00:00:00 2001 From: Winston Date: Thu, 3 Sep 2026 09:55:38 +1000 Subject: [PATCH 04/10] fix(interop): address CodeRabbit review on decode-bounds (LAB-2503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tool (tools/decode-bounds-reference.py): - `--require-extras` (same flag as wire-format-reference.py): a missing msgpack is a failure, so CI's optional-deps leg cannot pass without exercising the real decoder. CLI now fails closed on a flag typo, an unknown mode, two modes, or `generate --require-extras`. - MemoryError/RecursionError from unpackb fail the run naming the vector (the spec's failure_mode rule being violated), instead of a bare trace. - map32_max_claim_alone declared_slots counts pairs as two slots, matching nested_map16_depth_2048 and the field note. - Three new reject vectors, each pinning a distinct implementation error: array32_sum_wraps_u32 (running sum = 2^32 exactly, wraps to 0 in u32), map32_half_claim_wraps_u32_mul (per-header 2 x pairs = 2^32 wraps before the add), fixmap_short_by_one (one-slot-per-pair counting accepts it). msgpack-python 1.0.3 / 1.2.1 C / 1.2.1 pure-Python reject all three. - New mutation suite tools/test_decode_bounds_reference.py (11 guards) runs first in verify.yml, per the workflow's mutation-first doctrine. Spec: - interop-mode: map pair = two slots; per-header terms and the running sum MUST use >= 64-bit or checked/saturating arithmetic; overflow rejects. - wire-format: dropped the false "rmp-serde satisfies this inherently" claim (serde's Vec visitor pre-allocates from declared lengths); the envelope bytes are untrusted MessagePack too, so the decode-bounds pre-scan runs before StorageEnvelope is materialised (Retrieve Flow step 2, now in the read-side conformance list). - Matrix/changelog: decode-bounds vendoring is pending cachekit-py#276 and cachekit-rs#73 (both open), not done; footnote 16 lists the vector file. Rejected (replied on the thread): defining the SDK input-size cap at protocol level — that is a per-deployment sizing decision (LAB-2505). CodeRabbit-Resolved: .github/workflows/verify.yml:37:mutation suite CodeRabbit-Resolved: .github/workflows/verify.yml:46:optional-deps leg CodeRabbit-Resolved: sdk-feature-matrix.md:288:footnote 16 CodeRabbit-Resolved: spec/interop-mode.md:459:overflow-safe slot budget CodeRabbit-Resolved: spec/wire-format.md:380:rmp-serde inherently CodeRabbit-Resolved: spec/wire-format.md:384:envelope decode bounds CodeRabbit-Resolved: tools/decode-bounds-reference.py:83:map32 slot convention CodeRabbit-Resolved: tools/decode-bounds-reference.py:135:FBT001 --- .github/workflows/verify.yml | 3 +- CHANGELOG.md | 8 +- sdk-feature-matrix.md | 6 +- spec/interop-mode.md | 17 +++-- spec/wire-format.md | 20 ++--- test-vectors/decode-bounds.json | 58 ++++++++++++-- tools/decode-bounds-reference.py | 71 ++++++++++++----- tools/test_decode_bounds_reference.py | 105 ++++++++++++++++++++++++++ 8 files changed, 242 insertions(+), 46 deletions(-) create mode 100644 tools/test_decode_bounds_reference.py diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index 578b620..b202d49 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -30,6 +30,7 @@ jobs: # Mutation suite first, same doctrine as the version-floor guard below: # prove the fail-closed guards still fail before trusting the verify. python3 tools/test_wire_format_reference.py + python3 tools/test_decode_bounds_reference.py python3 tools/interop-reference.py verify python3 tools/interop-v2-reference.py verify python3 tools/encryption-verify.py @@ -43,7 +44,7 @@ jobs: python3 tools/interop-v2-reference.py verify python3 tools/encryption-verify.py --require-seal python3 tools/wire-format-reference.py verify --require-extras - python3 tools/decode-bounds-reference.py verify + python3 tools/decode-bounds-reference.py verify --require-extras - name: JS cross-check (independent encoder + @noble/hashes + WebCrypto) run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 76de96d..5fdbd8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,10 +11,12 @@ All notable changes to the CacheKit Protocol Specification. what the input can back (Σ declared slots ≤ input bytes − 1), and MUST fail closed with a catchable error. Follow-up to the LAB-2487 measurements. - New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) - (10 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py); - vendored and CI-executed by cachekit-py and cachekit-rs. + (13 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py) + and its mutation suite; to be vendored and CI-executed by + [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) and + [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (both open). - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) - cross-references the rules for the payload inside the envelope. + cross-references the rules for the envelope bytes and the payload inside them. - The single shared depth value stays [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item. diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 9d8bf09..ef8c4a2 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -280,12 +280,12 @@ its spec: | Requirement | Python | Rust | TypeScript | PHP | | :--- | :---: | :---: | :---: | :---: | | Key generation (Blake2b) | ✅ Compliant | N/A auto mode¹⁴ — interop/v1 keygen ✅ merged ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); `#[cachekit]` mints interop keys ([#35](https://github.com/cachekit-io/cachekit-rs/pull/35)) | ✅ Compliant | ⚠️ Untested | -| Wire format (ByteStorage) | ✅ Compliant¹⁵ | ✅ Canonical (`cachekit-core`) — unused for stored values¹⁵ | ✅ Compliant | ⚠️ Untested | +| Wire format (ByteStorage) | ✅ Compliant¹⁵ — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ✅ Canonical (`cachekit-core`) — unused for stored values¹⁵ | ✅ Compliant — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Untested | | Storage container (auto mode)¹⁵ | CK v3 frame (Python-internal) | Plain MessagePack (`rmp` named) — no envelope | Bare ByteStorage envelope (default) | — | | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) + decode-bounds (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) + decode-bounds (LAB-2503) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)), `decode-bounds.json` not yet vendored | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed pending [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed pending [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)), `decode-bounds.json` not yet vendored | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] @@ -293,7 +293,7 @@ its spec: > > ¹⁵ Auto-mode **stored bytes** are SDK-internal and differ per SDK — see [wire-format.md → SDK Storage Containers](spec/wire-format.md#sdk-storage-containers-auto-mode). Python stores the ByteStorage envelope *inside* its CK v3 frame; `cachekit-rs` does not use the envelope for values at all (it uses `cachekit-core` only for encryption). Cross-SDK value compatibility is exclusively an [interop-mode](spec/interop-mode.md) property (protocol#11). > -> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). +> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). > > ¹⁷ Version cells are **floors** (`X+`), not snapshots — they stay true as new versions publish; check the registry for the current release. Python's floor is the first *installable* one: interop merged under the `v0.13.0` tag, but neither `0.12.0` nor `0.13.0` was ever published to PyPI, so `0.14.0` is the earliest PyPI release containing interop mode. Do not "correct" this to 0.13.0 from the cachekit-py changelog alone. diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 2fcc7ca..14acc14 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -457,6 +457,11 @@ therefore: 2. **Never pre-allocate beyond what the input can back.** Every declared element or byte needs at least one input byte, so a structurally incomplete document (Σ declared slots > input bytes − 1) MUST be rejected *without* materialising it. + A map pair counts as two slots (key + value). Every per-header term and the running + sum MUST be computed in at least 64 bits or with checked/saturating arithmetic, and + an overflow is itself a rejection: two `array32` headers already exceed 2³², and a + 32-bit accumulator that wraps to a small value passes the budget + (`array32_sum_wraps_u32` and `map32_half_claim_wraps_u32_mul` pin the shapes). Do not assume a decoder is lazy: `rmp-serde` reads str/bin lazily but serde's `Vec` visitor still pre-allocates up to 1 MiB per collection from the declared length. A header-only structural walk before decoding (the pre-scan in @@ -469,7 +474,7 @@ These bounds are SDK-owned invariants, not library defaults: each SDK pins them explicitly and regression-tests them, so a decoder dependency bump cannot silently re-open the amplifier. [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the -bytes every decoder MUST reject (10) and MUST accept (2); the same rules apply to +bytes every decoder MUST reject (13) and MUST accept (2); the same rules apply to any other untrusted MessagePack decode in an SDK (auto-mode payloads after the envelope is unwrapped, invalidation events). The 40× residual — a *legal* payload still materialises far more than its byte size in language objects — is bounded by @@ -510,13 +515,15 @@ not re-litigated by accident. | `error_vectors` | 9 | Inputs that MUST be rejected (NaN, +Inf and −Inf as independent vectors, int overflow/underflow, naive datetime, bad segments incl. trailing newline). The `error` text is a maintainer note, not a normative message | [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) (see -[Decode bounds](#decode-bounds)) adds 10 `reject_vectors` (nested-header bombs, -over-claiming `array32`/`map32`/`bin32`/`str32` headers, a truncated array) and 2 +[Decode bounds](#decode-bounds)) adds 13 `reject_vectors` (nested-header bombs, +over-claiming `array32`/`map32`/`bin32`/`str32` headers, a truncated array and map, +two shapes that wrap 32-bit slot arithmetic) and 2 `accept_vectors` (32-deep nesting, a fully backed `array16`) that every SDK's untrusted decoder MUST honour; `tools/decode-bounds-reference.py verify` checks the file against its recipes and, when `msgpack-python` is installed, that the real -decoder rejects/accepts each vector (rejection only — the allocation rule is each -SDK's own guard). +decoder rejects/accepts each vector (rejection only — msgpack-python's default limits +reject them, which proves each vector trips a stock decoder's limits; each SDK's explicit bound and +no-pre-allocation guard are tested in that SDK, not here). Inputs use a tagged-JSON convention (`{"$set": …}`, `{"$float": "2.0"}`, `{"$int": "…"}`, `{"$datetime": "…"}`, `{"$uuid": "…"}`, `{"$bytes": ""}`) diff --git a/spec/wire-format.md b/spec/wire-format.md index cebffe1..57a8222 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -264,9 +264,9 @@ bytes are therefore - A conforming reader MUST decompress every pinned vector's `compressed_data` to its pinned input, **and MUST enforce [Retrieve Flow](#retrieve-flow) steps - 4, 5 and 9 while doing so.** Read-side conformance is not "the vectors pass": + 2, 4, 5 and 9 while doing so.** Read-side conformance is not "the vectors pass": every pinned vector is well-formed and declares a truthful `original_size`, so - they evidence **none** of those bounds, and a reader that omits all three + they evidence **none** of those bounds, and a reader that omits all four decompresses all of them successfully. The vectors prove decode interoperability; the bounds in [Security Limits](#security-limits) are a separate, non-negotiable obligation that no fixture can demonstrate. @@ -376,12 +376,13 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes(); > Additionally, a decoder MUST validate any declared MessagePack `bin`/array > length header against the remaining input bytes **before** allocating for it — > a 5-byte `bin32` header can otherwise declare a 4 GiB allocation from a -> ~30-byte envelope. (Slice-based decoders such as `rmp-serde` satisfy this -> inherently; readers that pre-allocate from length fields must check.) The -> payload *inside* the envelope is untrusted MessagePack too — decode it under -> the depth and allocation rules in -> [interop-mode.md → Decode bounds](interop-mode.md#decode-bounds), pinned by -> `test-vectors/decode-bounds.json`. +> ~30-byte envelope. No decoder satisfies this inherently — even slice-based +> ones pre-allocate collections from declared lengths. The envelope bytes *and* +> the payload inside them +> are both untrusted MessagePack — decode each under the depth and allocation +> rules in [interop-mode.md → Decode bounds](interop-mode.md#decode-bounds), +> pinned by `test-vectors/decode-bounds.json`, running the structural pre-scan +> before materialising `StorageEnvelope`. | Limit | Value | Purpose | | :--- | ---: | :--- | @@ -443,7 +444,8 @@ Input: raw_data (bytes), format (string, default "msgpack") Input: envelope_bytes 1. Validate: envelope_bytes.length <= 512 MiB -2. Deserialize: envelope = msgpack_decode(envelope_bytes) as StorageEnvelope +2. Deserialize: pre-scan envelope_bytes (decode bounds, see Security Limits), then + envelope = msgpack_decode(envelope_bytes) as StorageEnvelope // accept BOTH element[0] encodings: bin AND array-of-ints 3. Validate: envelope.compressed_data.length <= 512 MiB 4. Validate: envelope.original_size <= 512 MiB diff --git a/test-vectors/decode-bounds.json b/test-vectors/decode-bounds.json index 6a58520..21b9180 100644 --- a/test-vectors/decode-bounds.json +++ b/test-vectors/decode-bounds.json @@ -2,16 +2,16 @@ "version": "1.0.0", "spec": "spec/interop-mode.md#decode-bounds", "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", - "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, auto-mode payloads after the ByteStorage envelope is unwrapped, invalidation events. The bytes are plain MessagePack with no envelope.", + "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, the ByteStorage envelope bytes before StorageEnvelope is materialised, auto-mode payloads after the envelope is unwrapped, invalidation events. The bytes are plain MessagePack with no envelope.", "rules": { "depth": "Readers MUST bound nesting depth. The bound MUST be >= 32 and MUST be <= 1024; every reject vector tagged 'depth' nests deeper than 1024.", - "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining input can back (each element or byte needs >= 1 input byte), and MUST reject a structurally incomplete document. Every reject vector tagged 'overclaim' has declared_slots > input_len - 1 (the root header is the only byte that is not an element).", + "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining input can back (each element or byte needs >= 1 input byte), and MUST reject a structurally incomplete document. Every reject vector tagged 'overclaim' has declared_slots > input_len - 1 (the root header is the only byte that is not an element). A map pair counts as two slots (key + value). Every per-header term and the running sum MUST be computed in >= 64 bits or with checked/saturating arithmetic; an overflow is itself a rejection.", "failure_mode": "Rejection MUST surface as a catchable decode error that the SDK read path turns into a cache miss (fail-closed), never an uncaught crash or an OOM abort." }, "field_notes": { "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", - "declared_slots": "sum of every header's declared element/byte count (a nested header counts as one element of its parent)", + "declared_slots": "sum of every header's declared element/byte count; a map pair counts as two slots (key + value); a nested header counts as one element of its parent", "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message" }, "reject_vectors": [ @@ -116,7 +116,7 @@ }, { "name": "map32_max_claim_alone", - "description": "A lone 5-byte map32 header claiming 2^32-1 pairs.", + "description": "A lone 5-byte map32 header claiming 2^32-1 pairs (2^33-2 slots: each pair is a key and a value).", "construction": { "repeat_hex": "dfffffffff", "count": 1, @@ -125,7 +125,55 @@ "input_hex": "dfffffffff", "input_len": 5, "nesting_depth": 1, - "declared_slots": 4294967295, + "declared_slots": 8589934590, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "array32_sum_wraps_u32", + "description": "array32 claiming 2^32-1 elements whose first element is an array32 claiming 1: the declared slots sum to exactly 2^32, which a 32-bit accumulator wraps to 0 and then passes the slot budget.", + "construction": { + "repeat_hex": "ddffffffff", + "count": 1, + "suffix_hex": "dd00000001" + }, + "input_hex": "ddffffffffdd00000001", + "input_len": 10, + "nesting_depth": 2, + "declared_slots": 4294967296, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "map32_half_claim_wraps_u32_mul", + "description": "A lone map32 header claiming 2^31 pairs: the per-header term 2 x pairs is exactly 2^32, which a 32-bit multiply wraps to 0 before it is ever added to the budget.", + "construction": { + "repeat_hex": "df80000000", + "count": 1, + "suffix_hex": "" + }, + "input_hex": "df80000000", + "input_len": 5, + "nesting_depth": 1, + "declared_slots": 4294967296, + "reject_reasons": [ + "overclaim" + ] + }, + { + "name": "fixmap_short_by_one", + "description": "fixmap claiming 1 pair with the key present and the value missing: the map twin of fixarray_short_by_one. Counting one slot per pair (instead of two) accepts it.", + "construction": { + "repeat_hex": "81", + "count": 1, + "suffix_hex": "c0" + }, + "input_hex": "81c0", + "input_len": 2, + "nesting_depth": 1, + "declared_slots": 2, "reject_reasons": [ "overclaim" ] diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index 856304a..5451954 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -2,7 +2,7 @@ """Reference tool for test-vectors/decode-bounds.json (untrusted-decode bounds). Normative rules and the measurements behind them: spec/interop-mode.md → Decode -bounds. This file pins the bytes every SDK's decoder MUST reject (10) and MUST +bounds. This file pins the bytes every SDK's decoder MUST reject (13) and MUST accept (2, so the bound cannot over-tighten). Usage: @@ -10,7 +10,11 @@ that each vector's depth/slot tags match its arithmetic. When `msgpack` (msgpack-python) is importable, additionally checks the real decoder rejects every reject vector and accepts every accept - vector — rejection only; the allocation rule is each SDK's guard. + vector. Rejection only: msgpack-python's own default limits reject + them, which proves each vector trips a stock decoder's limits; each SDK's explicit bound + and no-pre-allocation guard are tested in that SDK, not here. + `--require-extras` turns a missing msgpack into a failure (CI's + optional-deps leg). generate Rewrites the vector file from the recipes below. """ @@ -79,8 +83,20 @@ def build() -> dict: "A lone 5-byte array32 header claiming 2^32-1 elements.", "dd" + u32(0xFFFFFFFF), 1, depth=1, slots=0xFFFFFFFF, reasons=["overclaim"]), recipe("map32_max_claim_alone", - "A lone 5-byte map32 header claiming 2^32-1 pairs.", - "df" + u32(0xFFFFFFFF), 1, depth=1, slots=0xFFFFFFFF, reasons=["overclaim"]), + "A lone 5-byte map32 header claiming 2^32-1 pairs (2^33-2 slots: each pair is a key and a value).", + "df" + u32(0xFFFFFFFF), 1, depth=1, slots=2 * 0xFFFFFFFF, reasons=["overclaim"]), + recipe("array32_sum_wraps_u32", + "array32 claiming 2^32-1 elements whose first element is an array32 claiming 1: the declared " + "slots sum to exactly 2^32, which a 32-bit accumulator wraps to 0 and then passes the slot budget.", + "dd" + u32(0xFFFFFFFF), 1, "dd" + u32(1), depth=2, slots=0xFFFFFFFF + 1, reasons=["overclaim"]), + recipe("map32_half_claim_wraps_u32_mul", + "A lone map32 header claiming 2^31 pairs: the per-header term 2 x pairs is exactly 2^32, which a " + "32-bit multiply wraps to 0 before it is ever added to the budget.", + "df" + u32(0x80000000), 1, depth=1, slots=2 * 0x80000000, reasons=["overclaim"]), + recipe("fixmap_short_by_one", + "fixmap claiming 1 pair with the key present and the value missing: the map twin of " + "fixarray_short_by_one. Counting one slot per pair (instead of two) accepts it.", + "81", 1, "c0", depth=1, slots=2, reasons=["overclaim"]), recipe("bin32_overclaim", "bin32 header claiming 2^32-1 bytes with 1 backing byte (a 6-byte document declaring a 4 GiB buffer).", "c6" + u32(0xFFFFFFFF), 1, "41", depth=0, slots=0xFFFFFFFF, reasons=["overclaim"]), @@ -107,9 +123,9 @@ def build() -> dict: "version": "1.0.0", "spec": "spec/interop-mode.md#decode-bounds", "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", - "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, auto-mode payloads after " - "the ByteStorage envelope is unwrapped, invalidation events. The bytes are plain MessagePack " - "with no envelope.", + "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, the ByteStorage envelope bytes " + "before StorageEnvelope is materialised, auto-mode payloads after the envelope is unwrapped, " + "invalidation events. The bytes are plain MessagePack with no envelope.", "rules": { "depth": f"Readers MUST bound nesting depth. The bound MUST be >= {MIN_DEPTH_FLOOR} and MUST be " f"<= {MAX_DEPTH_CEILING}; every reject vector tagged 'depth' nests deeper than " @@ -117,14 +133,17 @@ def build() -> dict: "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining " "input can back (each element or byte needs >= 1 input byte), and MUST reject a " "structurally incomplete document. Every reject vector tagged 'overclaim' has " - "declared_slots > input_len - 1 (the root header is the only byte that is not an element).", + "declared_slots > input_len - 1 (the root header is the only byte that is not an element). " + "A map pair counts as two slots (key + value). Every per-header term and the running sum " + "MUST be computed in >= 64 bits or with checked/saturating arithmetic; an overflow is " + "itself a rejection.", "failure_mode": "Rejection MUST surface as a catchable decode error that the SDK read path turns " "into a cache miss (fail-closed), never an uncaught crash or an OOM abort.", }, "field_notes": { "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", - "declared_slots": "sum of every header's declared element/byte count (a nested header counts as one element of its parent)", + "declared_slots": "sum of every header's declared element/byte count; a map pair counts as two slots (key + value); a nested header counts as one element of its parent", "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message", }, "reject_vectors": reject, @@ -132,13 +151,13 @@ def build() -> dict: } -def check(condition: bool, name: str, detail: str) -> None: +def check(condition: bool, name: str, detail: str) -> None: # noqa: FBT001 """Fail closed even under ``python -O`` (asserts would be stripped).""" if not condition: raise ValueError(f"{name}: {detail}") -def verify(document: dict) -> tuple[int, str]: +def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: fresh = build() check(document == fresh, "document", "vector file differs from the recipes; run `generate`") # input_hex / input_len are derived from `construction` by recipe(), so the equality @@ -152,41 +171,53 @@ def verify(document: dict) -> tuple[int, str]: if not reasons: check(v["nesting_depth"] <= MIN_DEPTH_FLOOR, v["name"], "accept vector deeper than the floor") + total = len(document["reject_vectors"]) + len(document["accept_vectors"]) try: import msgpack # type: ignore[import-not-found] except ImportError: - return len(document["reject_vectors"]) + len(document["accept_vectors"]), "stdlib only (msgpack absent)" + if require_extras: + raise ValueError("--require-extras set but msgpack is not importable") from None + return total, "stdlib only (msgpack absent)" for v in document["reject_vectors"]: data = bytes.fromhex(v["input_hex"]) try: msgpack.unpackb(data) - # Every msgpack-python unpack error subclasses ValueError (StackError, FormatError, - # ExtraData, max_*_len, incomplete input). Anything else — MemoryError, RecursionError, - # a crash — is the failure_mode rule being violated, so it must propagate, not count. + # unpackb surfaces every unpack failure as a ValueError (StackError, FormatError, + # ExtraData, max_*_len; it wraps OutOfData — the streaming Unpacker does not). Anything + # else is the failure_mode rule being violated, so it must fail the run, not count. except ValueError: continue + except (MemoryError, RecursionError) as e: + raise ValueError(f"{v['name']}: msgpack-python violated failure_mode ({type(e).__name__})") from e raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") for v in document["accept_vectors"]: msgpack.unpackb(bytes.fromhex(v["input_hex"])) - return len(document["reject_vectors"]) + len(document["accept_vectors"]), f"msgpack-python {msgpack.version} rejects/accepts as required" + return total, f"msgpack-python {msgpack.version} rejects/accepts as required" def main() -> None: - mode = sys.argv[1] if len(sys.argv) > 1 else "verify" + usage = f"usage: {sys.argv[0]} [verify|generate] [--require-extras]" + args = sys.argv[1:] + require_extras = "--require-extras" in args + modes = [a for a in args if a != "--require-extras"] + mode = modes[0] if modes else "verify" + # Fail closed on anything unexpected: a typo in the flag must not silently drop the + # extras requirement CI relies on, and generate has no extras to require. + if len(modes) > 1 or mode not in ("verify", "generate") or (require_extras and mode != "verify"): + sys.exit(usage) if mode == "generate": VECTORS.write_text(json.dumps(build(), indent=2) + "\n", encoding="utf-8") logging.info("wrote %s", VECTORS.relative_to(ROOT)) return - if mode != "verify": - sys.exit(f"usage: {sys.argv[0]} [verify|generate]") try: - count, leg = verify(json.loads(VECTORS.read_text(encoding="utf-8"))) + count, leg = verify(json.loads(VECTORS.read_text(encoding="utf-8")), require_extras=require_extras) except ValueError as e: sys.exit(f"decode-bounds verify FAILED: {e}") logging.info("decode-bounds: %d vectors OK (%s)", count, leg) if __name__ == "__main__": + # stdout, matching the pre-logging behaviour and the other tools' report lines. logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout) main() diff --git a/tools/test_decode_bounds_reference.py b/tools/test_decode_bounds_reference.py new file mode 100644 index 0000000..86e8b79 --- /dev/null +++ b/tools/test_decode_bounds_reference.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Mutation tests for decode-bounds-reference.py's fail-closed guards. + +Same doctrine as test_wire_format_reference.py: poison the input and watch each guard +fire, so a guard that degrades to always-pass is caught before `verify` is trusted. +Nothing here touches test-vectors/decode-bounds.json. + +Run: python3 tools/test_decode_bounds_reference.py (exit 1 on any failure) +""" + +from __future__ import annotations + +import copy +import importlib.util +import subprocess +import sys +import types +from collections.abc import Callable +from pathlib import Path +from unittest.mock import patch + +TOOL = Path(__file__).resolve().parent / "decode-bounds-reference.py" +SPEC = importlib.util.spec_from_file_location("dbr", TOOL) +dbr = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(dbr) + + +def expect_raises(name: str, fn: Callable[[], object], needle: str) -> str | None: + try: + fn() + except ValueError as e: + return None if needle in str(e) else f"{name}: raised but message lacks {needle!r}: {e}" + return f"{name}: did not raise" + + +def with_recipes(doc: dict) -> Callable[[], object]: + """Run verify() with build() patched to agree with `doc`, so only the tag checks stand.""" + def run() -> object: + with patch.object(dbr, "build", lambda: copy.deepcopy(doc)): + return dbr.verify(doc) + return run + + +def with_msgpack(unpackb: Callable[[bytes], object] | None, doc: dict, *, require_extras: bool = False) -> Callable[[], object]: + """Run verify() against a fake msgpack module (None = import blocked).""" + def run() -> object: + fake = None + if unpackb is not None: + fake = types.ModuleType("msgpack") + fake.unpackb, fake.version = unpackb, (0, 0, 0) + with patch.dict(sys.modules, {"msgpack": fake}): + return dbr.verify(doc, require_extras=require_extras) + return run + + +def raise_memory_error(_: bytes) -> None: + raise MemoryError + + +def cli_rejects(name: str, *args: str) -> str | None: + """The CLI must exit non-zero on anything it does not understand (fail closed).""" + rc = subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, check=False).returncode + return None if rc != 0 else f"{name}: exit 0 for {args}" + + +def main() -> None: + good = dbr.build() + results: list[str | None] = [] + + dbr.verify(good) # baseline: the real recipes pass + + drifted = copy.deepcopy(good) + drifted["reject_vectors"][0]["input_hex"] = "c0" + drifted["reject_vectors"][0]["input_hex"][2:] + results.append(expect_raises("file drift", lambda: dbr.verify(drifted), "differs from the recipes")) + + bad_depth = copy.deepcopy(good) + bad_depth["reject_vectors"][0]["nesting_depth"] = 5 # tagged 'depth' but no longer deeper than the ceiling + results.append(expect_raises("depth tag", with_recipes(bad_depth), "depth tag mismatch")) + + bad_slots = copy.deepcopy(good) + bad_slots["reject_vectors"][-1]["declared_slots"] = 1 # tagged 'overclaim' but no longer over-claims + results.append(expect_raises("overclaim tag", with_recipes(bad_slots), "overclaim tag mismatch")) + + deep_accept = copy.deepcopy(good) + deep_accept["accept_vectors"][0]["nesting_depth"] = dbr.MIN_DEPTH_FLOOR + 1 + results.append(expect_raises("accept floor", with_recipes(deep_accept), "deeper than the floor")) + + results.append(expect_raises("require-extras", with_msgpack(None, good, require_extras=True), "not importable")) + results.append(expect_raises("decoded reject", with_msgpack(lambda _: None, good), "decoded a reject vector")) + results.append(expect_raises("OOM not counted as reject", with_msgpack(raise_memory_error, good), "violated failure_mode")) + results.append(cli_rejects("flag typo", "verify", "--require-extra")) + results.append(cli_rejects("unknown mode", "bogus")) + results.append(cli_rejects("two modes", "verify", "generate")) + results.append(cli_rejects("generate with extras", "generate", "--require-extras")) + + failures = [f for f in results if f] + for f in failures: + print("FAIL", f, file=sys.stderr) + if failures: + sys.exit(1) + print(f"decode-bounds mutation suite: {len(results)} guards fire as required") + + +if __name__ == "__main__": + main() From b75adac4cc61f49edff6d84f1f77c8cd96af4b73 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 7 Sep 2026 09:32:21 +1000 Subject: [PATCH 05/10] fix(interop): address Kody round 2 on decode-bounds (LAB-2503) - tools/test_decode_bounds_reference.py: report line via logging (same stdout, message-only handler decode-bounds-reference.py installs); failures exit through sys.exit(str) -- stderr + exit 1, the tool's own fatal path -- so no print() remains and the failure stream is unchanged. - spec/wire-format.md Retrieve Flow step 2: `as StorageEnvelope` read as a TypeScript-style unchecked cast; the pseudo-code now uses the typed declaration cachekit-core actually ships (`let envelope: StorageEnvelope = rmp_serde::from_slice(...)`) and states that wrong arity or element type is a decode error that rejects. The element[0] dual-read line is unchanged. Expert panel (high stakes) on this delta: SHIP; its two tightenings are what is committed here. --- spec/wire-format.md | 3 ++- tools/test_decode_bounds_reference.py | 9 +++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/spec/wire-format.md b/spec/wire-format.md index 57a8222..cec0b60 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -445,7 +445,8 @@ Input: envelope_bytes 1. Validate: envelope_bytes.length <= 512 MiB 2. Deserialize: pre-scan envelope_bytes (decode bounds, see Security Limits), then - envelope = msgpack_decode(envelope_bytes) as StorageEnvelope + envelope: StorageEnvelope = msgpack_decode(envelope_bytes) + // typed decode, not a cast: wrong arity or element type -> Reject // accept BOTH element[0] encodings: bin AND array-of-ints 3. Validate: envelope.compressed_data.length <= 512 MiB 4. Validate: envelope.original_size <= 512 MiB diff --git a/tools/test_decode_bounds_reference.py b/tools/test_decode_bounds_reference.py index 86e8b79..dbb0231 100644 --- a/tools/test_decode_bounds_reference.py +++ b/tools/test_decode_bounds_reference.py @@ -12,6 +12,7 @@ import copy import importlib.util +import logging import subprocess import sys import types @@ -94,12 +95,12 @@ def main() -> None: results.append(cli_rejects("generate with extras", "generate", "--require-extras")) failures = [f for f in results if f] - for f in failures: - print("FAIL", f, file=sys.stderr) if failures: - sys.exit(1) - print(f"decode-bounds mutation suite: {len(results)} guards fire as required") + sys.exit("\n".join(f"FAIL {f}" for f in failures)) # stderr + exit 1, the tool's own fatal path + logging.info("decode-bounds mutation suite: %d guards fire as required", len(results)) if __name__ == "__main__": + # stdout, message-only: the same handler decode-bounds-reference.py installs. + logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout) main() From df24789938badd79f5735cd1b49383448afd32c5 Mon Sep 17 00:00:00 2001 From: Winston Date: Sun, 13 Sep 2026 16:14:53 +1000 Subject: [PATCH 06/10] docs(matrix): refresh decode-bounds CI status for py/rs/ts; noqa TRY003 on the reference tool (LAB-2503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - "Test vectors in CI" row: cachekit-py#276 and cachekit-rs#73 merged 2026-09-13. Rust default CI is green on main. Python default CI is red on 3.10/3.11 because the vendored suite's own test_nesting_ceiling oracle recurses (json.loads at depth 1024; LAB-3480) — stated in the cell rather than hidden behind a tick. TypeScript: cachekit-ts#121 vendors and executes the file (LAB-2737); "not yet vendored" dropped. - CHANGELOG: py/rs are no longer "both open"; the ts PR is listed. - tools/decode-bounds-reference.py: # noqa: TRY003 on the four ValueError raises, matching the file's existing noqa convention. This repo has no ruff config and verify.yml does not run ruff; the annotation is for reviewers' ruff, not a CI gate. --- CHANGELOG.md | 5 +++-- sdk-feature-matrix.md | 2 +- tools/decode-bounds-reference.py | 8 ++++---- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5fdbd8c..e104d2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,9 +12,10 @@ All notable changes to the CacheKit Protocol Specification. closed with a catchable error. Follow-up to the LAB-2487 measurements. - New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) (13 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py) - and its mutation suite; to be vendored and CI-executed by + and its mutation suite; vendored and CI-executed by [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) and - [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (both open). + [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (merged 2026-09-13) and + [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (open). - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) cross-references the rules for the envelope bytes and the payload inside them. - The single shared depth value stays [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index ef8c4a2..bd0e413 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -285,7 +285,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed pending [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed pending [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)), `decode-bounds.json` not yet vendored | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ default CI red on 3.10/3.11 since that merge: a test-oracle `RecursionError` in the same suite, not a vector failure (LAB-3480) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed pending [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737) | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index 5451954..42d0d13 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -154,7 +154,7 @@ def build() -> dict: def check(condition: bool, name: str, detail: str) -> None: # noqa: FBT001 """Fail closed even under ``python -O`` (asserts would be stripped).""" if not condition: - raise ValueError(f"{name}: {detail}") + raise ValueError(f"{name}: {detail}") # noqa: TRY003 def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: @@ -176,7 +176,7 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: import msgpack # type: ignore[import-not-found] except ImportError: if require_extras: - raise ValueError("--require-extras set but msgpack is not importable") from None + raise ValueError("--require-extras set but msgpack is not importable") from None # noqa: TRY003 return total, "stdlib only (msgpack absent)" for v in document["reject_vectors"]: @@ -189,8 +189,8 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: except ValueError: continue except (MemoryError, RecursionError) as e: - raise ValueError(f"{v['name']}: msgpack-python violated failure_mode ({type(e).__name__})") from e - raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") + raise ValueError(f"{v['name']}: msgpack-python violated failure_mode ({type(e).__name__})") from e # noqa: TRY003 + raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") # noqa: TRY003 for v in document["accept_vectors"]: msgpack.unpackb(bytes.fromhex(v["input_hex"])) return total, f"msgpack-python {msgpack.version} rejects/accepts as required" From 3695aa4698eafe198c26c7031dac29e7d3553541 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 27 Sep 2026 22:59:55 +1000 Subject: [PATCH 07/10] fix(interop): whole-document slot rule, three discriminating vectors, walk-derived tags (LAB-2503) decode-bounds.json (16 reject + 2 accept): - nested_array16_each_header_fits_sum_overclaims: every header fits the bytes after it and nesting is below the floor, so only the whole-document sum rejects it. - array32_sum_wraps_u32_small_first: passes a 32-bit running sum checked after every add, which array32_sum_wraps_u32 does not exercise. - nested_fixarray_depth_1025_complete: one level past the depth ceiling. decode-bounds-reference.py derives nesting_depth and declared_slots with a header-only structural walk instead of trusting the hand-entered tags, checks the set still contains each discriminating shape, and names the failure when a decoder rejects an accept vector. The generate --require-extras refusal is gone. interop-mode.md defines depth, states the whole-document rule and that per-header checks do not satisfy it, corrects the msgpack-python figure (10 KB -> ~82 MB), replaces the claim that each SDK's input-size cap bounds the legal-payload residual with the measured 72x on the decode input and no normative cap, names check_msgpack_structure, and adds two SHOULDs: test the bound on the smallest supported stack, and drive the vectors through the SDK's own guard. wire-format.md: Security Limits states the whole-document rule, the Verification Flow pre-scans before it decodes, and the no-fixture sentence is scoped to wire-format.json. Matrix: cachekit-ts#121 is merged, the stale cachekit-py CI caveat is dropped, cachekit-core ByteStorage is marked partial (its retrieve has no step-2 pre-scan), and footnote 16 notes the SDKs vendor the 13-reject version. --- CHANGELOG.md | 18 ++-- sdk-feature-matrix.md | 6 +- spec/interop-mode.md | 79 ++++++++------ spec/wire-format.md | 43 ++++---- test-vectors/decode-bounds.json | 52 ++++++++- tools/decode-bounds-reference.py | 145 +++++++++++++++++++++++--- tools/test_decode_bounds_reference.py | 46 ++++++-- 7 files changed, 305 insertions(+), 84 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fb3756..6e4c40d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,15 +11,17 @@ All notable changes to the CacheKit Protocol Specification. what the input can back (Σ declared slots ≤ input bytes − 1), and MUST fail closed with a catchable error. Follow-up to the LAB-2487 measurements. - New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) - (13 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py) - and its mutation suite; vendored and CI-executed by - [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276), - [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (both merged 2026-09-13) - and [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (merged 2026-09-20). + (16 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), + which derives every vector's depth and slot tags with a structural walk, and its + mutation suite. The SDKs vendor it; see the matrix's "Test vectors in CI" row. - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) - cross-references the rules for the envelope bytes and the payload inside them. -- The single shared depth value stays [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s - open item. + states the whole-document slot rule (per-header checks do not satisfy it) for the + envelope bytes and the payload inside them, and the Verification Flow pre-scans + before it decodes. +- Matrix: `cachekit-core` ByteStorage is ⚠️ — its `retrieve` has no step-2 pre-scan. +- Open: the shared depth value, and any cap on the ~70× materialisation of *legal* + payloads, stay [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s + items. ### Encryption — default-tenant conformance vector (LAB-4666) diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 818bed2..6278614 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -293,12 +293,12 @@ its spec: | Requirement | Python | Rust | TypeScript | PHP | | :--- | :---: | :---: | :---: | :---: | | Key generation (Blake2b) | ✅ Compliant | N/A auto mode¹⁴ — interop/v1 keygen ✅ merged ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); `#[cachekit]` mints interop keys ([#35](https://github.com/cachekit-io/cachekit-rs/pull/35)) | ✅ Compliant | ⚠️ Untested | -| Wire format (ByteStorage) | ✅ Compliant¹⁵ — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ✅ Canonical (`cachekit-core`) — unused for stored values¹⁵ | ✅ Compliant — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Untested | +| Wire format (ByteStorage) | ✅ Compliant¹⁵ — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Canonical (`cachekit-core`) — unused for stored values¹⁵; `ByteStorage::retrieve` decodes the envelope (typed `rmp_serde::from_slice` into `StorageEnvelope`) without the [Retrieve Flow](spec/wire-format.md#retrieve-flow) step-2 pre-scan | ✅ Compliant — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Untested | | Storage container (auto mode)¹⁵ | CK v3 frame (Python-internal) | Plain MessagePack (`rmp` named) — no envelope | Bare ByteStorage envelope (default) | — | | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ default CI red on 3.10/3.11 since that merge: a test-oracle `RecursionError` in the same suite, not a vector failure (LAB-3480) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed pending [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737) | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737) | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] @@ -306,7 +306,7 @@ its spec: > > ¹⁵ Auto-mode **stored bytes** are SDK-internal and differ per SDK — see [wire-format.md → SDK Storage Containers](spec/wire-format.md#sdk-storage-containers-auto-mode). Python stores the ByteStorage envelope *inside* its CK v3 frame; `cachekit-rs` does not use the envelope for values at all (it uses `cachekit-core` only for encryption). Cross-SDK value compatibility is exclusively an [interop-mode](spec/interop-mode.md) property (protocol#11). > -> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). +> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. The SDKs vendor `decode-bounds.json` at its 13-reject version: the three reject vectors added since (whole-document slot sum, small-first 32-bit wrap, depth 1025) run only in this repo until each SDK re-vendors. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). > > ¹⁷ Version cells are **floors** (`X+`), not snapshots — they stay true as new versions publish; check the registry for the current release. Python's floor is the first *installable* one: interop merged under the `v0.13.0` tag, but neither `0.12.0` nor `0.13.0` was ever published to PyPI, so `0.14.0` is the earliest PyPI release containing interop mode. Do not "correct" this to 0.13.0 from the cachekit-py changelog alone. diff --git a/spec/interop-mode.md b/spec/interop-mode.md index d4db43a..2d1fb7d 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -450,40 +450,62 @@ Interop values are read from a backend the SDK does not control, so every decode is an untrusted-input parser. A MessagePack collection header costs 1–5 bytes but may declare up to 2³²−1 elements, and an eager decoder pre-allocates the container *before* decoding its children; depth-first decoding stacks those allocations, so a -few KB of nested headers can drive hundreds of MB of transient heap (measured -15 KB → ~400 MB in `@msgpack/msgpack` 3.1.3; 10 KB → 67 MB in `msgpack-python` -1.2.1 with `array32` headers claiming `len(input)` elements). A reader MUST -therefore: - -1. **Bound nesting depth.** The bound MUST be at least 32 and MUST NOT exceed 1024. +few KB of nested headers can drive hundreds of MB of transient heap. Measured peak +heap: 15 KB → ~400 MB in `@msgpack/msgpack` 3.1.3, and 10 KB → ~82 MB in +`msgpack-python` 1.2.1 with `array32` headers claiming `len(input)` elements (8 bytes +× 1024 levels × input length: it allocates every level until its nesting limit trips). +A reader MUST therefore: + +1. **Bound nesting depth.** Depth is the number of collection headers on the + deepest path from the root; str, bin and scalars add nothing, so `[[null]]` has + depth 2. The bound MUST be at least 32 and MUST NOT exceed 1024. (Today: TypeScript 100, Rust 100, Python 1024. A single shared value is [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item; - until it is ratified, writers SHOULD keep values within 32 levels.) + until it is ratified, writers SHOULD keep values within 32 levels.) A recursive + native decoder can exhaust its thread stack below 1024 levels (`rmp-serde` in a + debug build does on a 2 MiB thread), so each SDK SHOULD test a complete document + at its own bound on its smallest supported stack. 2. **Never pre-allocate beyond what the input can back.** Every declared element or - byte needs at least one input byte, so a structurally incomplete document - (Σ declared slots > input bytes − 1) MUST be rejected *without* materialising it. - A map pair counts as two slots (key + value). Every per-header term and the running - sum MUST be computed in at least 64 bits or with checked/saturating arithmetic, and - an overflow is itself a rejection: two `array32` headers already exceed 2³², and a - 32-bit accumulator that wraps to a small value passes the budget - (`array32_sum_wraps_u32` and `map32_half_claim_wraps_u32_mul` pin the shapes). - Do not assume a decoder is lazy: `rmp-serde` reads str/bin lazily but serde's - `Vec` visitor still pre-allocates up to 1 MiB per collection from the + byte needs at least one input byte, so the declared slots summed over the whole + document MUST NOT exceed input bytes − 1, and a document that exceeds it MUST be + rejected *without* materialising it. Checking each header only against the input + that remains after it does not satisfy this: nested headers can each fit what + follows them while together declaring far more than the input holds + (`nested_array16_each_header_fits_sum_overclaims`). A map pair counts as two slots + (key + value). Exceeding the sum is sufficient to reject, but it is not the + definition of an incomplete document: `92 dc 00 00` sums to 2 and is still + truncated. Every per-header term and the running sum MUST be computed in at least + 64 bits or with checked/saturating arithmetic, and an overflow is itself a + rejection: two `array32` headers already exceed 2³², and a 32-bit accumulator that + wraps to a small value passes the budget (`array32_sum_wraps_u32`, + `array32_sum_wraps_u32_small_first` and `map32_half_claim_wraps_u32_mul` pin the + shapes). Do not assume a decoder is lazy: `rmp-serde` reads str/bin lazily but + serde's `Vec` visitor still pre-allocates up to 1 MiB per collection from the declared length. A header-only structural walk before decoding (the pre-scan in - `cachekit-ts`, `Unpacker.skip()` in `cachekit-py`, `check_structure` in + `cachekit-ts`, `check_msgpack_structure` in `cachekit-py`, `check_structure` in `cachekit-rs`) is sufficient. 3. **Fail closed, catchably.** Rejection surfaces as a decode error the SDK read path turns into a cache miss — never an uncaught crash or an OOM abort. These bounds are SDK-owned invariants, not library defaults: each SDK pins them explicitly and regression-tests them, so a decoder dependency bump cannot silently -re-open the amplifier. +re-open the amplifier. The vector verdicts alone cannot show that: a stock decoder's +default limits reject every reject vector today, so an SDK's test SHOULD drive the +vectors through its own guard (call the pre-scan directly, or bound peak memory) +rather than only assert that a decode fails. [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the -bytes every decoder MUST reject (13) and MUST accept (2); the same rules apply to +bytes every decoder MUST reject (16) and MUST accept (2); the same rules apply to any other untrusted MessagePack decode in an SDK (auto-mode payloads after the -envelope is unwrapped, invalidation events). The 40× residual — a *legal* payload -still materialises far more than its byte size in language objects — is bounded by -each SDK's input-size cap, not by these rules. +envelope is unwrapped, invalidation events). + +These rules do not bound the residual. A *legal*, fully backed document still +materialises far more memory than its size in language objects: an `array32` of +empty maps peaks at 72× its size in `msgpack-python` 1.2.1 (2 MB → 144 MB). The +ratio applies to the decode input, which for an auto-mode payload is the +LZ4-decompressed bytes (up to 512 MiB under +[wire-format.md → Security Limits](wire-format.md#security-limits)), not the stored +bytes. No normative input-size or element-count cap exists; a shared value belongs +with the depth value on [protocol#20](https://github.com/cachekit-io/protocol/issues/20). --- @@ -519,16 +541,9 @@ not re-litigated by accident. | `encryption_vectors` | 1 | Full HKDF-SHA256 → AES-256-GCM round-trip over plain-msgpack plaintext with the interop AAD (fixed nonce; decrypt-verified) | | `error_vectors` | 9 | Inputs that MUST be rejected (NaN, +Inf and −Inf as independent vectors, int overflow/underflow, naive datetime, bad segments incl. trailing newline). The `error` text is a maintainer note, not a normative message | -[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) (see -[Decode bounds](#decode-bounds)) adds 13 `reject_vectors` (nested-header bombs, -over-claiming `array32`/`map32`/`bin32`/`str32` headers, a truncated array and map, -two shapes that wrap 32-bit slot arithmetic) and 2 -`accept_vectors` (32-deep nesting, a fully backed `array16`) that every SDK's -untrusted decoder MUST honour; `tools/decode-bounds-reference.py verify` checks the -file against its recipes and, when `msgpack-python` is installed, that the real -decoder rejects/accepts each vector (rejection only — msgpack-python's default limits -reject them, which proves each vector trips a stock decoder's limits; each SDK's explicit bound and -no-pre-allocation guard are tested in that SDK, not here). +[`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the +[Decode bounds](#decode-bounds); `tools/decode-bounds-reference.py verify` checks it, +and the tool's docstring states what each CI leg proves. Inputs use a tagged-JSON convention (`{"$set": …}`, `{"$float": "2.0"}`, `{"$int": "…"}`, `{"$datetime": "…"}`, `{"$uuid": "…"}`, `{"$bytes": ""}`) diff --git a/spec/wire-format.md b/spec/wire-format.md index 3c1297a..db9c0d8 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -269,7 +269,9 @@ bytes are therefore they evidence **none** of those bounds, and a reader that omits all four decompresses all of them successfully. The vectors prove decode interoperability; the bounds in [Security Limits](#security-limits) are a - separate, non-negotiable obligation that no fixture can demonstrate. + separate, non-negotiable obligation that no `wire-format.json` vector + demonstrates. Step 2's decode bounds have their own fixture, + [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json). - A writer **other than the canonical `lz4_flex` writer** is NOT required to reproduce the pinned compressed bytes, and MUST NOT be judged non-conforming because its compressor output differs from the fixture — validate such a @@ -358,13 +360,14 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes(); ### Verification Flow ``` -1. Deserialize envelope from MessagePack -2. Validate security limits (see below) -3. Decompress compressed_data using original_size as size hint -4. Compute xxh3_64(decompressed_data) as big-endian 8 bytes -5. Compare with checksum field -6. If mismatch → reject (integrity failure) -7. Verify decompressed_data.length == original_size +1. Pre-scan the envelope bytes (decode bounds, see Security Limits below) +2. Deserialize envelope from MessagePack +3. Validate the size and ratio limits (see below) +4. Decompress compressed_data using original_size as size hint +5. Compute xxh3_64(decompressed_data) as big-endian 8 bytes +6. Compare with checksum field +7. If mismatch → reject (integrity failure) +8. Verify decompressed_data.length == original_size ``` --- @@ -373,16 +376,20 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes(); > [!IMPORTANT] > All three limits below MUST be enforced by every implementation of the ByteStorage envelope. The decompression bomb check uses integer arithmetic — do not substitute floating-point. -> Additionally, a decoder MUST validate any declared MessagePack `bin`/array -> length header against the remaining input bytes **before** allocating for it — -> a 5-byte `bin32` header can otherwise declare a 4 GiB allocation from a -> ~30-byte envelope. No decoder satisfies this inherently — even slice-based -> ones pre-allocate collections from declared lengths. The envelope bytes *and* -> the payload inside them -> are both untrusted MessagePack — decode each under the depth and allocation -> rules in [interop-mode.md → Decode bounds](interop-mode.md#decode-bounds), -> pinned by `test-vectors/decode-bounds.json`, running the structural pre-scan -> before materialising `StorageEnvelope`. +> Additionally, a decoder MUST NOT allocate for declared MessagePack lengths +> (collection, `bin`, `str`) more than the input can back: the declared slots, +> summed over the **whole document**, MUST NOT exceed the input length minus one, +> checked **before** anything is materialised. A 5-byte `bin32` header can +> otherwise declare a 4 GiB allocation from a ~30-byte envelope. Checking each +> header against the remaining input bytes does not satisfy this: nested headers +> can each fit what follows them while together declaring far more than the input +> holds. No decoder satisfies it inherently for collections — `rmp-serde` reads +> str/bin lazily, but serde's `Vec` visitor pre-allocates from declared +> lengths. The envelope bytes *and* the payload inside them are both untrusted +> MessagePack — decode each under the depth and allocation rules in +> [interop-mode.md → Decode bounds](interop-mode.md#decode-bounds) (which defines +> the slot count), pinned by `test-vectors/decode-bounds.json`, running the +> structural pre-scan before materialising `StorageEnvelope`. | Limit | Value | Purpose | | :--- | ---: | :--- | diff --git a/test-vectors/decode-bounds.json b/test-vectors/decode-bounds.json index 21b9180..64c3a44 100644 --- a/test-vectors/decode-bounds.json +++ b/test-vectors/decode-bounds.json @@ -5,7 +5,7 @@ "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, the ByteStorage envelope bytes before StorageEnvelope is materialised, auto-mode payloads after the envelope is unwrapped, invalidation events. The bytes are plain MessagePack with no envelope.", "rules": { "depth": "Readers MUST bound nesting depth. The bound MUST be >= 32 and MUST be <= 1024; every reject vector tagged 'depth' nests deeper than 1024.", - "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining input can back (each element or byte needs >= 1 input byte), and MUST reject a structurally incomplete document. Every reject vector tagged 'overclaim' has declared_slots > input_len - 1 (the root header is the only byte that is not an element). A map pair counts as two slots (key + value). Every per-header term and the running sum MUST be computed in >= 64 bits or with checked/saturating arithmetic; an overflow is itself a rejection.", + "overclaim": "Readers MUST NOT pre-allocate beyond what the input can back (each element or byte needs >= 1 input byte): declared slots summed over the whole document MUST NOT exceed input_len - 1, checked before anything is materialised. Checking each header against the remaining input does not satisfy this. Readers MUST reject a structurally incomplete document. Every reject vector tagged 'overclaim' has declared_slots > input_len - 1 (the root header is the only byte that is not an element). A map pair counts as two slots (key + value). Every per-header term and the running sum MUST be computed in >= 64 bits or with checked/saturating arithmetic; an overflow is itself a rejection.", "failure_mode": "Rejection MUST surface as a catchable decode error that the SDK read path turns into a cache miss (fail-closed), never an uncaught crash or an OOM abort." }, "field_notes": { @@ -66,6 +66,22 @@ "overclaim" ] }, + { + "name": "nested_array16_each_header_fits_sum_overclaims", + "description": "30 nested array16 headers each claiming 2000 elements, then 2000 nils. Every header fits the bytes that follow it and the nesting is below the depth floor, so only the sum over the whole document (60 000 > input_len - 1) rejects it. A per-header check (claim <= remaining input) accepts it and lets an eager decoder pre-allocate 30 x 2000 slots.", + "construction": { + "repeat_hex": "dc07d0", + "count": 30, + "suffix_hex": "c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0" + }, + "input_hex": "dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0dc07d0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0", + "input_len": 2090, + "nesting_depth": 30, + "declared_slots": 60000, + "reject_reasons": [ + "overclaim" + ] + }, { "name": "nested_fixarray_depth_2048_complete", "description": "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", @@ -82,6 +98,22 @@ "depth" ] }, + { + "name": "nested_fixarray_depth_1025_complete", + "description": "Structurally COMPLETE document nested 1025 deep, one level past the ceiling: only the depth bound rejects it, and a reader whose bound exceeds 1024 accepts it.", + "construction": { + "repeat_hex": "91", + "count": 1025, + "suffix_hex": "c0" + }, + "input_hex": "9191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191c0", + "input_len": 1026, + "nesting_depth": 1025, + "declared_slots": 1025, + "reject_reasons": [ + "depth" + ] + }, { "name": "array16_overclaim_shallow", "description": "One array16 header claiming 10 000 elements with 3 backing bytes.", @@ -132,7 +164,7 @@ }, { "name": "array32_sum_wraps_u32", - "description": "array32 claiming 2^32-1 elements whose first element is an array32 claiming 1: the declared slots sum to exactly 2^32, which a 32-bit accumulator wraps to 0 and then passes the slot budget.", + "description": "array32 claiming 2^32-1 elements whose first element is an array32 claiming 1: the declared slots sum to exactly 2^32, which a 32-bit accumulator checked only once the sum is complete wraps to 0 and passes. One checked after every add rejects it at the first header; see array32_sum_wraps_u32_small_first.", "construction": { "repeat_hex": "ddffffffff", "count": 1, @@ -146,6 +178,22 @@ "overclaim" ] }, + { + "name": "array32_sum_wraps_u32_small_first", + "description": "fixarray claiming 1 element that is an array32 claiming 2^32-1: the running sum is 1, then exactly 2^32, so a 32-bit accumulator checked after every add sees 1 and then 0 and passes both checks.", + "construction": { + "repeat_hex": "91", + "count": 1, + "suffix_hex": "ddffffffff" + }, + "input_hex": "91ddffffffff", + "input_len": 6, + "nesting_depth": 2, + "declared_slots": 4294967296, + "reject_reasons": [ + "overclaim" + ] + }, { "name": "map32_half_claim_wraps_u32_mul", "description": "A lone map32 header claiming 2^31 pairs: the per-header term 2 x pairs is exactly 2^32, which a 32-bit multiply wraps to 0 before it is ever added to the budget.", diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index 42d0d13..d6390ce 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -2,12 +2,15 @@ """Reference tool for test-vectors/decode-bounds.json (untrusted-decode bounds). Normative rules and the measurements behind them: spec/interop-mode.md → Decode -bounds. This file pins the bytes every SDK's decoder MUST reject (13) and MUST -accept (2, so the bound cannot over-tighten). +bounds. This file pins the bytes every SDK's decoder MUST reject and MUST accept +(so the bound cannot over-tighten). Usage: - verify (default) stdlib-only. Checks the file equals the recipes below and - that each vector's depth/slot tags match its arithmetic. When + verify (default) stdlib-only. Checks the file equals the recipes below, + derives each vector's depth and declared slots with a header-only + structural walk (never trusting the hand-entered tags), checks the + reject reasons against them, and checks the set still contains the + shapes that separate a conforming reader from a near miss. When `msgpack` (msgpack-python) is importable, additionally checks the real decoder rejects every reject vector and accepts every accept vector. Rejection only: msgpack-python's own default limits reject @@ -20,6 +23,7 @@ from __future__ import annotations +from collections.abc import Callable import json import logging from pathlib import Path @@ -28,9 +32,6 @@ ROOT = Path(__file__).resolve().parents[1] VECTORS = ROOT / "test-vectors" / "decode-bounds.json" -# Every SDK's current depth bound is <= MAX_DEPTH_CEILING (ts 100, rs 100, py -# 1024), so a reject vector nested deeper than this is rejected by all of them; -# every SDK accepts at least MIN_DEPTH_FLOOR levels. MAX_DEPTH_CEILING = 1024 MIN_DEPTH_FLOOR = 32 @@ -72,10 +73,21 @@ def build() -> dict: recipe("nested_map16_depth_2048", "Map twin of nested_array16_depth_2048 (map pre-allocation is typically larger per slot).", "de" + u16(2000), 2048, depth=2048, slots=2048 * 2 * 2000, reasons=["depth", "overclaim"]), + recipe("nested_array16_each_header_fits_sum_overclaims", + "30 nested array16 headers each claiming 2000 elements, then 2000 nils. Every header fits the bytes " + "that follow it and the nesting is below the depth floor, so only the sum over the whole document " + "(60 000 > input_len - 1) rejects it. A per-header check (claim <= remaining input) accepts it and " + "lets an eager decoder pre-allocate 30 x 2000 slots.", + "dc" + u16(2000), 30, "c0" * 2000, depth=30, slots=30 * 2000, reasons=["overclaim"]), recipe("nested_fixarray_depth_2048_complete", "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, " "so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", "91", 2048, "c0", depth=2048, slots=2048, reasons=["depth"]), + recipe("nested_fixarray_depth_1025_complete", + "Structurally COMPLETE document nested 1025 deep, one level past the ceiling: only the depth bound " + "rejects it, and a reader whose bound exceeds 1024 accepts it.", + "91", MAX_DEPTH_CEILING + 1, "c0", depth=MAX_DEPTH_CEILING + 1, slots=MAX_DEPTH_CEILING + 1, + reasons=["depth"]), recipe("array16_overclaim_shallow", "One array16 header claiming 10 000 elements with 3 backing bytes.", "dc" + u16(10000), 1, "010203", depth=1, slots=10000, reasons=["overclaim"]), @@ -87,8 +99,14 @@ def build() -> dict: "df" + u32(0xFFFFFFFF), 1, depth=1, slots=2 * 0xFFFFFFFF, reasons=["overclaim"]), recipe("array32_sum_wraps_u32", "array32 claiming 2^32-1 elements whose first element is an array32 claiming 1: the declared " - "slots sum to exactly 2^32, which a 32-bit accumulator wraps to 0 and then passes the slot budget.", + "slots sum to exactly 2^32, which a 32-bit accumulator checked only once the sum is complete wraps " + "to 0 and passes. One checked after every add rejects it at the first header; see " + "array32_sum_wraps_u32_small_first.", "dd" + u32(0xFFFFFFFF), 1, "dd" + u32(1), depth=2, slots=0xFFFFFFFF + 1, reasons=["overclaim"]), + recipe("array32_sum_wraps_u32_small_first", + "fixarray claiming 1 element that is an array32 claiming 2^32-1: the running sum is 1, then exactly " + "2^32, so a 32-bit accumulator checked after every add sees 1 and then 0 and passes both checks.", + "91", 1, "dd" + u32(0xFFFFFFFF), depth=2, slots=1 + 0xFFFFFFFF, reasons=["overclaim"]), recipe("map32_half_claim_wraps_u32_mul", "A lone map32 header claiming 2^31 pairs: the per-header term 2 x pairs is exactly 2^32, which a " "32-bit multiply wraps to 0 before it is ever added to the budget.", @@ -130,9 +148,11 @@ def build() -> dict: "depth": f"Readers MUST bound nesting depth. The bound MUST be >= {MIN_DEPTH_FLOOR} and MUST be " f"<= {MAX_DEPTH_CEILING}; every reject vector tagged 'depth' nests deeper than " f"{MAX_DEPTH_CEILING}.", - "overclaim": "Readers MUST NOT pre-allocate for a collection/str/bin header more than the remaining " - "input can back (each element or byte needs >= 1 input byte), and MUST reject a " - "structurally incomplete document. Every reject vector tagged 'overclaim' has " + "overclaim": "Readers MUST NOT pre-allocate beyond what the input can back (each element or byte needs " + ">= 1 input byte): declared slots summed over the whole document MUST NOT exceed " + "input_len - 1, checked before anything is materialised. Checking each header against the " + "remaining input does not satisfy this. Readers MUST reject a structurally incomplete " + "document. Every reject vector tagged 'overclaim' has " "declared_slots > input_len - 1 (the root header is the only byte that is not an element). " "A map pair counts as two slots (key + value). Every per-header term and the running sum " "MUST be computed in >= 64 bits or with checked/saturating arithmetic; an overflow is " @@ -151,6 +171,81 @@ def build() -> dict: } +# type byte -> (kind, bytes in its length field or fixed payload); fix* types are handled in walk(). +_WIDE = { + 0xC4: ("bytes", 1), 0xC5: ("bytes", 2), 0xC6: ("bytes", 4), + 0xD9: ("bytes", 1), 0xDA: ("bytes", 2), 0xDB: ("bytes", 4), + 0xC7: ("ext", 1), 0xC8: ("ext", 2), 0xC9: ("ext", 4), + 0xDC: ("array", 2), 0xDD: ("array", 4), 0xDE: ("map", 2), 0xDF: ("map", 4), + 0xC0: ("fixed", 0), 0xC2: ("fixed", 0), 0xC3: ("fixed", 0), + 0xCA: ("fixed", 4), 0xCB: ("fixed", 8), + 0xCC: ("fixed", 1), 0xCD: ("fixed", 2), 0xCE: ("fixed", 4), 0xCF: ("fixed", 8), + 0xD0: ("fixed", 1), 0xD1: ("fixed", 2), 0xD2: ("fixed", 4), 0xD3: ("fixed", 8), + 0xD4: ("fixed", 2), 0xD5: ("fixed", 3), 0xD6: ("fixed", 5), 0xD7: ("fixed", 9), 0xD8: ("fixed", 17), +} + + +def walk(data: bytes) -> dict: + """Header-only structural walk, the reference for the `nesting_depth` and `declared_slots` tags. + + Reads headers in document order and skips str/bin/ext payloads; stops at the end of the + root item or of the input. Also reports two near-miss readers' verdicts for the coverage + guards: `per_header_fits` (every claim <= the bytes after its header) and `u32_sum_fits` + (a 32-bit running sum, checked after every add, never exceeds len - 1). + """ + pos = depth = slots = sum32 = 0 + per_header_fits = u32_sum_fits = True + owed: list[int] = [] # children still owed by each open collection + while pos < len(data): + t = data[pos] + pos += 1 + if t <= 0x7F or t >= 0xE0: + kind, width, n = "fixed", 0, 0 + elif t <= 0x8F: + kind, width, n = "map", 0, t & 0x0F + elif t <= 0x9F: + kind, width, n = "array", 0, t & 0x0F + elif t <= 0xBF: + kind, width, n = "bytes", 0, t & 0x1F + elif t in _WIDE: + kind, width = _WIDE[t] + n = 0 + else: + raise ValueError(f"walk: type byte {t:#04x} is never used") # noqa: TRY003 + if kind != "fixed" and width: + if pos + width > len(data): + break + n = int.from_bytes(data[pos:pos + width], "big") + pos += width + if kind == "fixed": + pos += width + else: + claim = 2 * n if kind == "map" else n + slots += claim + per_header_fits &= claim <= len(data) - pos + sum32 = (sum32 + claim) % 2**32 + u32_sum_fits &= sum32 <= len(data) - 1 + if kind in ("array", "map"): + depth = max(depth, len(owed) + 1) + if claim: + owed.append(claim) + continue + else: + pos += n + (kind == "ext") # payload (+ the ext type byte) + if pos > len(data): + break + while owed: # one item completed: settle every collection it finishes + owed[-1] -= 1 + if owed[-1]: + break + owed.pop() + if not owed: + return {"nesting_depth": depth, "declared_slots": slots, "complete": pos == len(data), + "per_header_fits": per_header_fits, "u32_sum_fits": u32_sum_fits} + return {"nesting_depth": depth, "declared_slots": slots, "complete": False, + "per_header_fits": per_header_fits, "u32_sum_fits": u32_sum_fits} + + def check(condition: bool, name: str, detail: str) -> None: # noqa: FBT001 """Fail closed even under ``python -O`` (asserts would be stripped).""" if not condition: @@ -161,8 +256,12 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: fresh = build() check(document == fresh, "document", "vector file differs from the recipes; run `generate`") # input_hex / input_len are derived from `construction` by recipe(), so the equality - # above already proves them; what still needs checking is the hand-entered tags. + # above already proves them; the hand-entered tags are checked against the walk. + walked = {} for v in document["reject_vectors"] + document["accept_vectors"]: + w = walked[v["name"]] = walk(bytes.fromhex(v["input_hex"])) + check(w["nesting_depth"] == v["nesting_depth"], v["name"], "nesting_depth differs from the walk") + check(w["declared_slots"] == v["declared_slots"], v["name"], "declared_slots differs from the walk") reasons = v.get("reject_reasons", []) check(("depth" in reasons) == (v["nesting_depth"] > MAX_DEPTH_CEILING), v["name"], "depth tag mismatch") # Slot budget: every declared element (including a nested header) costs >= 1 input @@ -170,6 +269,19 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: check(("overclaim" in reasons) == (v["declared_slots"] > v["input_len"] - 1), v["name"], "overclaim tag mismatch") if not reasons: check(v["nesting_depth"] <= MIN_DEPTH_FLOOR, v["name"], "accept vector deeper than the floor") + check(w["complete"], v["name"], "accept vector is not one complete document") + + # Coverage: each guard names a near-miss reader that would otherwise pass every verdict. + def some_reject(test: Callable[[dict, dict], bool]) -> bool: + return any(test(v, walked[v["name"]]) for v in document["reject_vectors"]) + check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["per_header_fits"] + and 2 <= v["nesting_depth"] < MIN_DEPTH_FLOOR), + "coverage", "no reject vector needs the whole-document sum (per-header checks pass, nesting below the floor)") + check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["u32_sum_fits"]), + "coverage", "no reject vector passes a 32-bit running sum checked after every add") + check(some_reject(lambda v, w: v["reject_reasons"] == ["depth"] and w["complete"] + and v["nesting_depth"] == MAX_DEPTH_CEILING + 1), + "coverage", "no complete reject vector sits one level past the depth ceiling") total = len(document["reject_vectors"]) + len(document["accept_vectors"]) try: @@ -192,7 +304,10 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: raise ValueError(f"{v['name']}: msgpack-python violated failure_mode ({type(e).__name__})") from e # noqa: TRY003 raise ValueError(f"{v['name']}: msgpack-python decoded a reject vector") # noqa: TRY003 for v in document["accept_vectors"]: - msgpack.unpackb(bytes.fromhex(v["input_hex"])) + try: + msgpack.unpackb(bytes.fromhex(v["input_hex"])) + except ValueError as e: + raise ValueError(f"{v['name']}: msgpack-python rejected an accept vector") from e # noqa: TRY003 return total, f"msgpack-python {msgpack.version} rejects/accepts as required" @@ -203,8 +318,8 @@ def main() -> None: modes = [a for a in args if a != "--require-extras"] mode = modes[0] if modes else "verify" # Fail closed on anything unexpected: a typo in the flag must not silently drop the - # extras requirement CI relies on, and generate has no extras to require. - if len(modes) > 1 or mode not in ("verify", "generate") or (require_extras and mode != "verify"): + # extras requirement CI relies on. + if len(modes) > 1 or mode not in ("verify", "generate"): sys.exit(usage) if mode == "generate": VECTORS.write_text(json.dumps(build(), indent=2) + "\n", encoding="utf-8") diff --git a/tools/test_decode_bounds_reference.py b/tools/test_decode_bounds_reference.py index dbb0231..9c6cf32 100644 --- a/tools/test_decode_bounds_reference.py +++ b/tools/test_decode_bounds_reference.py @@ -58,6 +58,10 @@ def raise_memory_error(_: bytes) -> None: raise MemoryError +def raise_value_error(_: bytes) -> None: + raise ValueError("stock limit") # noqa: TRY003 + + def cli_rejects(name: str, *args: str) -> str | None: """The CLI must exit non-zero on anything it does not understand (fail closed).""" rc = subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, check=False).returncode @@ -74,25 +78,55 @@ def main() -> None: drifted["reject_vectors"][0]["input_hex"] = "c0" + drifted["reject_vectors"][0]["input_hex"][2:] results.append(expect_raises("file drift", lambda: dbr.verify(drifted), "differs from the recipes")) + lied_depth = copy.deepcopy(good) + lied_depth["reject_vectors"][0]["nesting_depth"] = 5 # hand-entered tag disagrees with the bytes + results.append(expect_raises("depth vs walk", with_recipes(lied_depth), "nesting_depth differs from the walk")) + + lied_slots = copy.deepcopy(good) + lied_slots["reject_vectors"][-1]["declared_slots"] = 1 + results.append(expect_raises("slots vs walk", with_recipes(lied_slots), "declared_slots differs from the walk")) + bad_depth = copy.deepcopy(good) - bad_depth["reject_vectors"][0]["nesting_depth"] = 5 # tagged 'depth' but no longer deeper than the ceiling + bad_depth["reject_vectors"][0]["reject_reasons"] = ["overclaim"] # nests past the ceiling, untagged results.append(expect_raises("depth tag", with_recipes(bad_depth), "depth tag mismatch")) - bad_slots = copy.deepcopy(good) - bad_slots["reject_vectors"][-1]["declared_slots"] = 1 # tagged 'overclaim' but no longer over-claims - results.append(expect_raises("overclaim tag", with_recipes(bad_slots), "overclaim tag mismatch")) + complete = copy.deepcopy(good) # the truncated fixarray made whole, still tagged 'overclaim' + complete["reject_vectors"][-1] = dbr.recipe("fixarray_short_by_one", "", "95", 1, "c0" * 5, + depth=1, slots=5, reasons=["overclaim"]) + results.append(expect_raises("overclaim tag", with_recipes(complete), "overclaim tag mismatch")) deep_accept = copy.deepcopy(good) - deep_accept["accept_vectors"][0]["nesting_depth"] = dbr.MIN_DEPTH_FLOOR + 1 + deep_accept["accept_vectors"][0] = dbr.recipe("nested_fixarray_depth_33", "", "91", dbr.MIN_DEPTH_FLOOR + 1, "c0", + depth=dbr.MIN_DEPTH_FLOOR + 1, slots=dbr.MIN_DEPTH_FLOOR + 1, reasons=[]) + del deep_accept["accept_vectors"][0]["reject_reasons"] results.append(expect_raises("accept floor", with_recipes(deep_accept), "deeper than the floor")) + cut_accept = copy.deepcopy(good) + cut_accept["accept_vectors"][1] = dbr.recipe("array16_256_backed_nils", "", "dc" + dbr.u16(256), 1, "c0" * 255, + depth=1, slots=256, reasons=[]) + del cut_accept["accept_vectors"][1]["reject_reasons"] + results.append(expect_raises("accept complete", with_recipes(cut_accept), "not one complete document")) + + # Coverage: drop every vector that separates each near-miss reader and watch its guard fire. + def without(test: Callable[[dict, dict], bool]) -> dict: + doc = copy.deepcopy(good) + doc["reject_vectors"] = [v for v in doc["reject_vectors"] + if not test(v, dbr.walk(bytes.fromhex(v["input_hex"])))] + return doc + results.append(expect_raises("coverage: sum", with_recipes(without( + lambda v, w: w["per_header_fits"] and v["reject_reasons"] == ["overclaim"])), "whole-document sum")) + results.append(expect_raises("coverage: u32", with_recipes(without( + lambda v, w: w["u32_sum_fits"])), "32-bit running sum")) + results.append(expect_raises("coverage: depth", with_recipes(without( + lambda v, w: v["nesting_depth"] == dbr.MAX_DEPTH_CEILING + 1)), "one level past the depth ceiling")) + results.append(expect_raises("require-extras", with_msgpack(None, good, require_extras=True), "not importable")) results.append(expect_raises("decoded reject", with_msgpack(lambda _: None, good), "decoded a reject vector")) results.append(expect_raises("OOM not counted as reject", with_msgpack(raise_memory_error, good), "violated failure_mode")) + results.append(expect_raises("rejected accept", with_msgpack(raise_value_error, good), "rejected an accept vector")) results.append(cli_rejects("flag typo", "verify", "--require-extra")) results.append(cli_rejects("unknown mode", "bogus")) results.append(cli_rejects("two modes", "verify", "generate")) - results.append(cli_rejects("generate with extras", "generate", "--require-extras")) failures = [f for f in results if f] if failures: From 6fb7d20469714f7287fcd9e3a9d2cd159cbb2bf9 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 27 Sep 2026 23:42:19 +1000 Subject: [PATCH 08/10] fix(interop): guard-level conformance MUST, map-depth and ext vectors, self-pinning coverage (LAB-2503) A reject verdict does not show when a reader rejected: a reader with per-header checks alone rejects the incomplete slot-sum vector at end of input, after pre-allocating. interop-mode.md now makes it a MUST that an SDK's conformance test asserts its structural guard rejects each reject vector before anything is materialised (call the guard, or assert its distinguishing error), and a verdict-only run does not demonstrate conformance. The vector description and the coverage comment no longer claim verdict-level evidence. decode-bounds.json 1.1.0 (17 reject + 3 accept): - add nested_fixmap_depth_1025_complete (reject) and nested_fixmap_depth_32 (accept): every depth vector was an array, so array-only depth counting passed. - add ext32_overclaim; ext lengths join the normative slot list in both specs and in field_notes. - cut nested_fixarray_depth_2048_complete (dominated by the 1025 vector). - the fixture no longer carries an internal reference in a description. decode-bounds-reference.py: walk() reports array_depth and splits the 32-bit model into u32_add_fits and u32_mul_fits; one coverage check per near-miss guard (per-header, 32-bit add, 32-bit multiply, array spine, map spine), plus negative controls so a model that always passes is caught. walk.complete is documented as framing only. Mutation suite: each coverage test drops exactly one vector, each negative control forces one model to pass, and a hex table pins walk() per framing rule (50 guards; a 28-mutant sweep of walk() is fully killed). interop-mode.md also defines a map as one depth level and states that a structurally incomplete document MUST be rejected. Matrix: ByteStorage is partial in all three SDKs (each decodes the envelope through cachekit-core's retrieve, which has no step-2 pre-scan). The Python and Rust decode-bounds cells are partial until their tests assert the guard's rejection; TypeScript already does. Footnote 16 names each SDK's vendored revision. --- CHANGELOG.md | 19 ++-- sdk-feature-matrix.md | 6 +- spec/interop-mode.md | 27 +++--- spec/wire-format.md | 2 +- test-vectors/decode-bounds.json | 65 ++++++++++---- tools/decode-bounds-reference.py | 120 +++++++++++++++++--------- tools/test_decode_bounds_reference.py | 83 +++++++++++++++--- 7 files changed, 231 insertions(+), 91 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e4c40d..ce3de31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,15 +10,20 @@ All notable changes to the CacheKit Protocol Specification. readers MUST bound nesting depth (≥ 32, ≤ 1024), MUST NOT pre-allocate beyond what the input can back (Σ declared slots ≤ input bytes − 1), and MUST fail closed with a catchable error. Follow-up to the LAB-2487 measurements. -- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) - (16 reject + 2 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), +- New [`test-vectors/decode-bounds.json`](test-vectors/decode-bounds.json) `1.1.0` + (17 reject + 3 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), which derives every vector's depth and slot tags with a structural walk, and its - mutation suite. The SDKs vendor it; see the matrix's "Test vectors in CI" row. + mutation suite. The SDKs vendor earlier revisions; see the matrix's footnote 16. +- An SDK's conformance test MUST assert that its structural guard rejects each reject + vector before materialising it. A verdict alone does not show when a reader rejected. - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) - states the whole-document slot rule (per-header checks do not satisfy it) for the - envelope bytes and the payload inside them, and the Verification Flow pre-scans - before it decodes. -- Matrix: `cachekit-core` ByteStorage is ⚠️ — its `retrieve` has no step-2 pre-scan. + states the whole-document slot rule (per-header checks do not satisfy it; ext + lengths count) for the envelope bytes and the payload inside them, and the + Verification Flow pre-scans before it decodes. +- Matrix: ByteStorage is ⚠️ in all three SDKs, because each decodes the envelope with + `cachekit-core`'s `ByteStorage::retrieve`, which has no step-2 pre-scan. The + decode-bounds test cells for Python and Rust are ⚠️ until their tests assert the + guard's rejection. - Open: the shared depth value, and any cap on the ~70× materialisation of *legal* payloads, stay [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s items. diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 6278614..ae69915 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -293,12 +293,12 @@ its spec: | Requirement | Python | Rust | TypeScript | PHP | | :--- | :---: | :---: | :---: | :---: | | Key generation (Blake2b) | ✅ Compliant | N/A auto mode¹⁴ — interop/v1 keygen ✅ merged ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); `#[cachekit]` mints interop keys ([#35](https://github.com/cachekit-io/cachekit-rs/pull/35)) | ✅ Compliant | ⚠️ Untested | -| Wire format (ByteStorage) | ✅ Compliant¹⁵ — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Canonical (`cachekit-core`) — unused for stored values¹⁵; `ByteStorage::retrieve` decodes the envelope (typed `rmp_serde::from_slice` into `StorageEnvelope`) without the [Retrieve Flow](spec/wire-format.md#retrieve-flow) step-2 pre-scan | ✅ Compliant — envelope-bytes decode-bounds pre-scan (LAB-2503) pending verification | ⚠️ Untested | +| Wire format (ByteStorage) | ⚠️ Compliant¹⁵ except the envelope pre-scan: payload decodes are pre-scanned (`unpackb_bounded`), but the envelope goes through `cachekit-core`'s `ByteStorage::retrieve`, which has no [Retrieve Flow](spec/wire-format.md#retrieve-flow) step-2 pre-scan | ⚠️ Canonical (`cachekit-core`) — unused for stored values¹⁵; `ByteStorage::retrieve` decodes the envelope (typed `rmp_serde::from_slice` into `StorageEnvelope`) with no step-2 pre-scan | ⚠️ Compliant except the envelope pre-scan: payload decodes are pre-scanned, but the envelope goes through `cachekit-core`'s `ByteStorage::retrieve` (`cachekit-core-ts` `unpack`), which has no step-2 pre-scan | ⚠️ Untested | | Storage container (auto mode)¹⁵ | CK v3 frame (Python-internal) | Plain MessagePack (`rmp` named) — no envelope | Bare ByteStorage envelope (default) | — | | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737) | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting each vector's pre-scan error | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] @@ -306,7 +306,7 @@ its spec: > > ¹⁵ Auto-mode **stored bytes** are SDK-internal and differ per SDK — see [wire-format.md → SDK Storage Containers](spec/wire-format.md#sdk-storage-containers-auto-mode). Python stores the ByteStorage envelope *inside* its CK v3 frame; `cachekit-rs` does not use the envelope for values at all (it uses `cachekit-core` only for encryption). Cross-SDK value compatibility is exclusively an [interop-mode](spec/interop-mode.md) property (protocol#11). > -> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. The SDKs vendor `decode-bounds.json` at its 13-reject version: the three reject vectors added since (whole-document slot sum, small-first 32-bit wrap, depth 1025) run only in this repo until each SDK re-vendors. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). +> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)), and — since LAB-2503 — `decode-bounds.json` ([`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), `verify` in both the stdlib and the optional-deps legs) against reference implementations. The SDKs vendor earlier revisions of `decode-bounds.json`, all labelled `1.0.0`: cachekit-py and cachekit-ts the 13-reject / 2-accept revision, cachekit-rs a 10-reject / 2-accept one. This repo's file is `1.1.0` (17 reject, 3 accept); vectors newer than an SDK's copy run only here until that SDK re-vendors. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). > > ¹⁷ Version cells are **floors** (`X+`), not snapshots — they stay true as new versions publish; check the registry for the current release. Python's floor is the first *installable* one: interop merged under the `v0.13.0` tag, but neither `0.12.0` nor `0.13.0` was ever published to PyPI, so `0.14.0` is the earliest PyPI release containing interop mode. Do not "correct" this to 0.13.0 from the cachekit-py changelog alone. diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 2d1fb7d..87cb949 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -457,8 +457,9 @@ heap: 15 KB → ~400 MB in `@msgpack/msgpack` 3.1.3, and 10 KB → ~82 MB in A reader MUST therefore: 1. **Bound nesting depth.** Depth is the number of collection headers on the - deepest path from the root; str, bin and scalars add nothing, so `[[null]]` has - depth 2. The bound MUST be at least 32 and MUST NOT exceed 1024. + deepest path from the root. A map counts one level, like an array; str, bin, ext + and scalars add nothing, so `[[null]]` and `{"": [null]}` both have depth 2. The + bound MUST be at least 32 and MUST NOT exceed 1024. (Today: TypeScript 100, Rust 100, Python 1024. A single shared value is [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s open item; until it is ratified, writers SHOULD keep values within 32 levels.) A recursive @@ -466,15 +467,16 @@ A reader MUST therefore: debug build does on a 2 MiB thread), so each SDK SHOULD test a complete document at its own bound on its smallest supported stack. 2. **Never pre-allocate beyond what the input can back.** Every declared element or - byte needs at least one input byte, so the declared slots summed over the whole + byte (collection elements; str, bin and ext bytes) needs at least one input byte, + so the declared slots summed over the whole document MUST NOT exceed input bytes − 1, and a document that exceeds it MUST be rejected *without* materialising it. Checking each header only against the input that remains after it does not satisfy this: nested headers can each fit what follows them while together declaring far more than the input holds (`nested_array16_each_header_fits_sum_overclaims`). A map pair counts as two slots - (key + value). Exceeding the sum is sufficient to reject, but it is not the - definition of an incomplete document: `92 dc 00 00` sums to 2 and is still - truncated. Every per-header term and the running sum MUST be computed in at least + (key + value). Exceeding the sum is sufficient to reject but does not define an + incomplete document: `92 dc 00 00` sums to 2 and is still truncated. A reader MUST + reject a structurally incomplete document as well. Every per-header term and the running sum MUST be computed in at least 64 bits or with checked/saturating arithmetic, and an overflow is itself a rejection: two `array32` headers already exceed 2³², and a 32-bit accumulator that wraps to a small value passes the budget (`array32_sum_wraps_u32`, @@ -489,12 +491,15 @@ A reader MUST therefore: These bounds are SDK-owned invariants, not library defaults: each SDK pins them explicitly and regression-tests them, so a decoder dependency bump cannot silently -re-open the amplifier. The vector verdicts alone cannot show that: a stock decoder's -default limits reject every reject vector today, so an SDK's test SHOULD drive the -vectors through its own guard (call the pre-scan directly, or bound peak memory) -rather than only assert that a decode fails. +re-open the amplifier. A verdict cannot show that, because it does not say *when* a +reader rejected: a stock decoder's default limits reject every reject vector today, +and a reader with per-header checks alone rejects the incomplete ones at end of input, +after it has pre-allocated for them. An SDK's conformance test MUST therefore assert +that its structural guard rejects each reject vector before anything is materialised, +by calling the guard directly or by asserting the guard's distinguishing error. A run +that only asserts that a decode fails does not demonstrate conformance. [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the -bytes every decoder MUST reject (16) and MUST accept (2); the same rules apply to +bytes every decoder MUST reject (17) and MUST accept (3); the same rules apply to any other untrusted MessagePack decode in an SDK (auto-mode payloads after the envelope is unwrapped, invalidation events). diff --git a/spec/wire-format.md b/spec/wire-format.md index db9c0d8..b775a0b 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -377,7 +377,7 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes(); > [!IMPORTANT] > All three limits below MUST be enforced by every implementation of the ByteStorage envelope. The decompression bomb check uses integer arithmetic — do not substitute floating-point. > Additionally, a decoder MUST NOT allocate for declared MessagePack lengths -> (collection, `bin`, `str`) more than the input can back: the declared slots, +> (collection, `str`, `bin`, `ext`) more than the input can back: the declared slots, > summed over the **whole document**, MUST NOT exceed the input length minus one, > checked **before** anything is materialised. A 5-byte `bin32` header can > otherwise declare a 4 GiB allocation from a ~30-byte envelope. Checking each diff --git a/test-vectors/decode-bounds.json b/test-vectors/decode-bounds.json index 64c3a44..a995dad 100644 --- a/test-vectors/decode-bounds.json +++ b/test-vectors/decode-bounds.json @@ -1,5 +1,5 @@ { - "version": "1.0.0", + "version": "1.1.0", "spec": "spec/interop-mode.md#decode-bounds", "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, the ByteStorage envelope bytes before StorageEnvelope is materialised, auto-mode payloads after the envelope is unwrapped, invalidation events. The bytes are plain MessagePack with no envelope.", @@ -10,14 +10,14 @@ }, "field_notes": { "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", - "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", - "declared_slots": "sum of every header's declared element/byte count; a map pair counts as two slots (key + value); a nested header counts as one element of its parent", + "nesting_depth": "collection headers along the deepest spine; a map counts one level, like an array (str/bin/ext and scalars count as 0)", + "declared_slots": "sum of every header's declared element/byte count (collections, str, bin, ext; fixext declares none); a map pair counts as two slots (key + value); a nested header counts as one element of its parent", "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message" }, "reject_vectors": [ { "name": "nested_array16_depth_2048", - "description": "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The LAB-2487 amplifier shape: an eager decoder pre-allocates 2000 slots per level before hitting EOF. 2000 < input_len, so a per-collection cap of len(input) does NOT reject it.", + "description": "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The measured amplifier shape: an eager decoder pre-allocates 2000 slots per level before hitting EOF. 2000 < input_len, so a per-collection cap of len(input) does NOT reject it.", "construction": { "repeat_hex": "dc07d0", "count": 2048, @@ -68,7 +68,7 @@ }, { "name": "nested_array16_each_header_fits_sum_overclaims", - "description": "30 nested array16 headers each claiming 2000 elements, then 2000 nils. Every header fits the bytes that follow it and the nesting is below the depth floor, so only the sum over the whole document (60 000 > input_len - 1) rejects it. A per-header check (claim <= remaining input) accepts it and lets an eager decoder pre-allocate 30 x 2000 slots.", + "description": "30 nested array16 headers each claiming 2000 elements, then 2000 nils. Every header fits the bytes that follow it and the nesting is below the depth floor, so of the structural rules only the sum over the whole document (60 000 > input_len - 1) catches it. A reader with per-header checks alone pre-allocates 30 x 2000 slots and then rejects at end of input, so the verdict cannot tell the two apart: only an SDK test asserting its guard's rejection can.", "construction": { "repeat_hex": "dc07d0", "count": 30, @@ -83,33 +83,33 @@ ] }, { - "name": "nested_fixarray_depth_2048_complete", - "description": "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", + "name": "nested_fixarray_depth_1025_complete", + "description": "Structurally COMPLETE document nested 1025 deep, one level past the ceiling: only the depth bound rejects it, and a reader whose bound exceeds 1024 accepts it.", "construction": { "repeat_hex": "91", - "count": 2048, + "count": 1025, "suffix_hex": "c0" }, - "input_hex": "9191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191c0", - "input_len": 2049, - "nesting_depth": 2048, - "declared_slots": 2048, + "input_hex": "9191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191c0", + "input_len": 1026, + "nesting_depth": 1025, + "declared_slots": 1025, "reject_reasons": [ "depth" ] }, { - "name": "nested_fixarray_depth_1025_complete", - "description": "Structurally COMPLETE document nested 1025 deep, one level past the ceiling: only the depth bound rejects it, and a reader whose bound exceeds 1024 accepts it.", + "name": "nested_fixmap_depth_1025_complete", + "description": "Map twin of nested_fixarray_depth_1025_complete ({\"\": {\"\": ... null}}): a guard that counts depth on array headers only accepts it.", "construction": { - "repeat_hex": "91", + "repeat_hex": "81a0", "count": 1025, "suffix_hex": "c0" }, - "input_hex": "9191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191919191c0", - "input_len": 1026, + "input_hex": "81a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a0c0", + "input_len": 2051, "nesting_depth": 1025, - "declared_slots": 1025, + "declared_slots": 2050, "reject_reasons": [ "depth" ] @@ -258,6 +258,22 @@ "overclaim" ] }, + { + "name": "ext32_overclaim", + "description": "ext32 twin of bin32_overclaim (type 5, 1 backing byte): ext lengths count as slots too.", + "construction": { + "repeat_hex": "c9ffffffff", + "count": 1, + "suffix_hex": "0541" + }, + "input_hex": "c9ffffffff0541", + "input_len": 7, + "nesting_depth": 0, + "declared_slots": 4294967295, + "reject_reasons": [ + "overclaim" + ] + }, { "name": "fixarray_short_by_one", "description": "fixarray claiming 5 elements with 4 present: the minimal truncated document.", @@ -289,6 +305,19 @@ "nesting_depth": 32, "declared_slots": 32 }, + { + "name": "nested_fixmap_depth_32", + "description": "Map twin of nested_fixarray_depth_32: a map counts one level, and a pair two slots.", + "construction": { + "repeat_hex": "81a0", + "count": 32, + "suffix_hex": "c0" + }, + "input_hex": "81a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a081a0c0", + "input_len": 65, + "nesting_depth": 32, + "declared_slots": 64 + }, { "name": "array16_256_backed_nils", "description": "array16 header claiming 256 elements with all 256 present. A *16 header that is fully backed by input is legitimate; the allocation rule is about backing, not header width.", diff --git a/tools/decode-bounds-reference.py b/tools/decode-bounds-reference.py index d6390ce..11c1483 100644 --- a/tools/decode-bounds-reference.py +++ b/tools/decode-bounds-reference.py @@ -9,15 +9,16 @@ verify (default) stdlib-only. Checks the file equals the recipes below, derives each vector's depth and declared slots with a header-only structural walk (never trusting the hand-entered tags), checks the - reject reasons against them, and checks the set still contains the - shapes that separate a conforming reader from a near miss. When - `msgpack` (msgpack-python) is importable, additionally checks the - real decoder rejects every reject vector and accepts every accept - vector. Rejection only: msgpack-python's own default limits reject - them, which proves each vector trips a stock decoder's limits; each SDK's explicit bound - and no-pre-allocation guard are tested in that SDK, not here. - `--require-extras` turns a missing msgpack into a failure (CI's - optional-deps leg). + reject reasons against them, and checks the set still holds a vector + each named near-miss structural guard would pass. When `msgpack` + (msgpack-python) is importable, additionally checks the real decoder + rejects every reject vector and accepts every accept vector. + A verdict says nothing about WHEN a reader rejected: a stock decoder + rejects every reject vector by its own limits or at end of input, + possibly after pre-allocating. Whether an SDK's structural guard + rejects each vector before materialising it is asserted in that SDK + (spec: Decode bounds), not here. `--require-extras` turns a missing + msgpack into a failure (CI's optional-deps leg). generate Rewrites the vector file from the recipes below. """ @@ -62,7 +63,7 @@ def recipe(name: str, description: str, repeat_hex: str, count: int, suffix_hex: def build() -> dict: reject = [ recipe("nested_array16_depth_2048", - "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The LAB-2487 " + "2048 nested array16 headers each claiming 2000 elements, 0 backing bytes. The measured " "amplifier shape: an eager decoder pre-allocates 2000 slots per level before hitting EOF. " "2000 < input_len, so a per-collection cap of len(input) does NOT reject it.", "dc" + u16(2000), 2048, depth=2048, slots=2048 * 2000, reasons=["depth", "overclaim"]), @@ -75,19 +76,21 @@ def build() -> dict: "de" + u16(2000), 2048, depth=2048, slots=2048 * 2 * 2000, reasons=["depth", "overclaim"]), recipe("nested_array16_each_header_fits_sum_overclaims", "30 nested array16 headers each claiming 2000 elements, then 2000 nils. Every header fits the bytes " - "that follow it and the nesting is below the depth floor, so only the sum over the whole document " - "(60 000 > input_len - 1) rejects it. A per-header check (claim <= remaining input) accepts it and " - "lets an eager decoder pre-allocate 30 x 2000 slots.", + "that follow it and the nesting is below the depth floor, so of the structural rules only the sum " + "over the whole document (60 000 > input_len - 1) catches it. A reader with per-header checks alone " + "pre-allocates 30 x 2000 slots and then rejects at end of input, so the verdict cannot tell the two " + "apart: only an SDK test asserting its guard's rejection can.", "dc" + u16(2000), 30, "c0" * 2000, depth=30, slots=30 * 2000, reasons=["overclaim"]), - recipe("nested_fixarray_depth_2048_complete", - "Structurally COMPLETE document ([[...[null]...]]) nested 2048 deep: every header is backed, " - "so only the depth bound rejects it. Isolates the depth rule from the allocation rule.", - "91", 2048, "c0", depth=2048, slots=2048, reasons=["depth"]), recipe("nested_fixarray_depth_1025_complete", "Structurally COMPLETE document nested 1025 deep, one level past the ceiling: only the depth bound " "rejects it, and a reader whose bound exceeds 1024 accepts it.", "91", MAX_DEPTH_CEILING + 1, "c0", depth=MAX_DEPTH_CEILING + 1, slots=MAX_DEPTH_CEILING + 1, reasons=["depth"]), + recipe("nested_fixmap_depth_1025_complete", + "Map twin of nested_fixarray_depth_1025_complete ({\"\": {\"\": ... null}}): a guard that counts " + "depth on array headers only accepts it.", + "81a0", MAX_DEPTH_CEILING + 1, "c0", depth=MAX_DEPTH_CEILING + 1, slots=2 * (MAX_DEPTH_CEILING + 1), + reasons=["depth"]), recipe("array16_overclaim_shallow", "One array16 header claiming 10 000 elements with 3 backing bytes.", "dc" + u16(10000), 1, "010203", depth=1, slots=10000, reasons=["overclaim"]), @@ -121,6 +124,9 @@ def build() -> dict: recipe("str32_overclaim", "str32 twin of bin32_overclaim.", "db" + u32(0xFFFFFFFF), 1, "41", depth=0, slots=0xFFFFFFFF, reasons=["overclaim"]), + recipe("ext32_overclaim", + "ext32 twin of bin32_overclaim (type 5, 1 backing byte): ext lengths count as slots too.", + "c9" + u32(0xFFFFFFFF), 1, "0541", depth=0, slots=0xFFFFFFFF, reasons=["overclaim"]), recipe("fixarray_short_by_one", "fixarray claiming 5 elements with 4 present: the minimal truncated document.", "95", 1, "c0c0c0c0", depth=1, slots=5, reasons=["overclaim"]), @@ -130,6 +136,9 @@ def build() -> dict: "[[...[null]...]] nested 32 deep, complete. A conforming reader MUST accept it: the depth " "bound may not be tighter than 32.", "91", MIN_DEPTH_FLOOR, "c0", depth=MIN_DEPTH_FLOOR, slots=MIN_DEPTH_FLOOR, reasons=[]), + recipe("nested_fixmap_depth_32", + "Map twin of nested_fixarray_depth_32: a map counts one level, and a pair two slots.", + "81a0", MIN_DEPTH_FLOOR, "c0", depth=MIN_DEPTH_FLOOR, slots=2 * MIN_DEPTH_FLOOR, reasons=[]), recipe("array16_256_backed_nils", "array16 header claiming 256 elements with all 256 present. A *16 header that is fully " "backed by input is legitimate; the allocation rule is about backing, not header width.", @@ -138,7 +147,7 @@ def build() -> dict: for v in accept: del v["reject_reasons"] return { - "version": "1.0.0", + "version": "1.1.0", "spec": "spec/interop-mode.md#decode-bounds", "generator": "tools/decode-bounds-reference.py generate (CPython stdlib)", "scope": "Any untrusted MessagePack decode in any SDK: interop/v1 values, the ByteStorage envelope bytes " @@ -162,8 +171,11 @@ def build() -> dict: }, "field_notes": { "construction": "input = bytes.fromhex(repeat_hex) * count + bytes.fromhex(suffix_hex)", - "nesting_depth": "collection headers along the deepest spine (str/bin count as 0)", - "declared_slots": "sum of every header's declared element/byte count; a map pair counts as two slots (key + value); a nested header counts as one element of its parent", + "nesting_depth": "collection headers along the deepest spine; a map counts one level, like an array " + "(str/bin/ext and scalars count as 0)", + "declared_slots": "sum of every header's declared element/byte count (collections, str, bin, ext; fixext " + "declares none); a map pair counts as two slots (key + value); a nested header counts " + "as one element of its parent", "reject_reasons": "which rule(s) the vector violates; a maintainer note, not a normative message", }, "reject_vectors": reject, @@ -189,13 +201,20 @@ def walk(data: bytes) -> dict: """Header-only structural walk, the reference for the `nesting_depth` and `declared_slots` tags. Reads headers in document order and skips str/bin/ext payloads; stops at the end of the - root item or of the input. Also reports two near-miss readers' verdicts for the coverage - guards: `per_header_fits` (every claim <= the bytes after its header) and `u32_sum_fits` - (a 32-bit running sum, checked after every add, never exceeds len - 1). + root item or of the input. `complete` is framing only (one root item, nothing owed, no + trailing bytes): it does not check str UTF-8 or ext contents, so `a1ff` is complete. + + Also reports what four near-miss structural guards conclude, for the coverage checks: + `per_header_fits` (every claim <= the bytes after its header), `u32_add_fits` (a 32-bit + running sum checked after every add; a term past 2^32 - 1 does not fit it), `u32_mul_fits` + (the map term 2 x pairs computed in 32 bits, summed exactly) and `array_depth` (depth + counted on array headers only). """ - pos = depth = slots = sum32 = 0 - per_header_fits = u32_sum_fits = True - owed: list[int] = [] # children still owed by each open collection + budget = len(data) - 1 + pos = depth = array_depth = arrays_open = slots = sum_add32 = sum_mul = 0 + per_header_fits = u32_add_fits = u32_mul_fits = True + complete = False + owed: list[list[int]] = [] # [children still owed, 1 if array] per open collection while pos < len(data): t = data[pos] pos += 1 @@ -223,27 +242,35 @@ def walk(data: bytes) -> dict: claim = 2 * n if kind == "map" else n slots += claim per_header_fits &= claim <= len(data) - pos - sum32 = (sum32 + claim) % 2**32 - u32_sum_fits &= sum32 <= len(data) - 1 + if claim < 2**32: + sum_add32 = (sum_add32 + claim) % 2**32 + u32_add_fits &= sum_add32 <= budget + else: + u32_add_fits = False + sum_mul += claim % 2**32 + u32_mul_fits &= sum_mul <= budget if kind in ("array", "map"): + is_array = int(kind == "array") depth = max(depth, len(owed) + 1) + array_depth = max(array_depth, arrays_open + is_array) if claim: - owed.append(claim) + owed.append([claim, is_array]) + arrays_open += is_array continue else: pos += n + (kind == "ext") # payload (+ the ext type byte) if pos > len(data): break while owed: # one item completed: settle every collection it finishes - owed[-1] -= 1 - if owed[-1]: + owed[-1][0] -= 1 + if owed[-1][0]: break - owed.pop() + arrays_open -= owed.pop()[1] if not owed: - return {"nesting_depth": depth, "declared_slots": slots, "complete": pos == len(data), - "per_header_fits": per_header_fits, "u32_sum_fits": u32_sum_fits} - return {"nesting_depth": depth, "declared_slots": slots, "complete": False, - "per_header_fits": per_header_fits, "u32_sum_fits": u32_sum_fits} + complete = pos == len(data) + break + return {"nesting_depth": depth, "declared_slots": slots, "complete": complete, "array_depth": array_depth, + "per_header_fits": per_header_fits, "u32_add_fits": u32_add_fits, "u32_mul_fits": u32_mul_fits} def check(condition: bool, name: str, detail: str) -> None: # noqa: FBT001 @@ -271,17 +298,28 @@ def verify(document: dict, *, require_extras: bool = False) -> tuple[int, str]: check(v["nesting_depth"] <= MIN_DEPTH_FLOOR, v["name"], "accept vector deeper than the floor") check(w["complete"], v["name"], "accept vector is not one complete document") - # Coverage: each guard names a near-miss reader that would otherwise pass every verdict. + # Coverage: the set holds a vector each named near-miss STRUCTURAL GUARD would pass. These are + # guard-level facts from the walk. A decoder may still reject the same bytes later, at end of + # input, so they bind only an SDK test that asserts its guard rejected (spec: Decode bounds). def some_reject(test: Callable[[dict, dict], bool]) -> bool: return any(test(v, walked[v["name"]]) for v in document["reject_vectors"]) check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["per_header_fits"] and 2 <= v["nesting_depth"] < MIN_DEPTH_FLOOR), - "coverage", "no reject vector needs the whole-document sum (per-header checks pass, nesting below the floor)") - check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["u32_sum_fits"]), + "coverage", "no reject vector that per-header checks pass, nested below the depth floor") + check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["u32_add_fits"]), "coverage", "no reject vector passes a 32-bit running sum checked after every add") + check(some_reject(lambda v, w: v["reject_reasons"] == ["overclaim"] and w["u32_mul_fits"]), + "coverage", "no reject vector passes a map term computed in 32 bits") + check(some_reject(lambda v, w: v["reject_reasons"] == ["depth"] and w["complete"] + and w["array_depth"] == MAX_DEPTH_CEILING + 1), + "coverage", "no complete array spine one level past the depth ceiling") check(some_reject(lambda v, w: v["reject_reasons"] == ["depth"] and w["complete"] - and v["nesting_depth"] == MAX_DEPTH_CEILING + 1), - "coverage", "no complete reject vector sits one level past the depth ceiling") + and v["nesting_depth"] == MAX_DEPTH_CEILING + 1 and w["array_depth"] <= MAX_DEPTH_CEILING), + "coverage", "no complete map spine one level past the depth ceiling") + # Negative controls: each near-miss model must also reject something, or the guards above are vacuous. + for name, flag in (("array16_overclaim_shallow", "per_header_fits"), ("array32_sum_wraps_u32", "u32_add_fits"), + ("array32_sum_wraps_u32", "u32_mul_fits")): + check(name in walked and not walked[name][flag], "coverage", f"{flag} passes {name}: the model is vacuous") total = len(document["reject_vectors"]) + len(document["accept_vectors"]) try: diff --git a/tools/test_decode_bounds_reference.py b/tools/test_decode_bounds_reference.py index 9c6cf32..5603c6f 100644 --- a/tools/test_decode_bounds_reference.py +++ b/tools/test_decode_bounds_reference.py @@ -62,6 +62,57 @@ def raise_value_error(_: bytes) -> None: raise ValueError("stock limit") # noqa: TRY003 +# hex -> (nesting_depth, declared_slots, complete, array_depth): one row per framing rule walk() implements. +WALK_TABLE = { + "05": (0, 0, True, 0), # positive fixint + "7f": (0, 0, True, 0), # last positive fixint, not a fixmap + "e0": (0, 0, True, 0), # negative fixint + "a3616263": (0, 3, True, 0), # fixstr: length in the type byte + "a1ff": (0, 1, True, 0), # framing only: invalid UTF-8 is still complete + "b0" + "41" * 16: (0, 16, True, 0), # fixstr mask 0x1f + "d90141": (0, 1, True, 0), # str8 + "c403010203": (0, 3, True, 0), # bin8 + "c702054142": (0, 2, True, 0), # ext8: length, type byte, payload + "d40100": (0, 0, True, 0), # fixext1 declares no slots + "d801" + "00" * 16: (0, 0, True, 0), # fixext16 + "cf" + "00" * 8: (0, 0, True, 0), # uint64 + "ca00000000": (0, 0, True, 0), # float32 + "cf00": (0, 0, False, 0), # fixed payload cut short + "90": (1, 0, True, 1), # empty array still counts a level + "81a0c0": (1, 2, True, 0), # fixmap: one level, two slots + "8f": (1, 30, False, 0), # fixmap mask 0x0f + "9181a0c0": (2, 3, True, 1), # map inside array: array_depth counts arrays only + "9291c091c0": (2, 4, True, 2), # sibling arrays: a closed array leaves the count + "92dc0000": (2, 2, False, 2), # truncated with the sum inside the budget + "dc00": (0, 0, False, 0), # length field cut short + "c0c0": (0, 0, False, 0), # trailing byte: not one document +} + + +# hex -> (per_header_fits, u32_add_fits, u32_mul_fits): the near-miss models on their boundaries. +FLAG_TABLE = { + "92c0c0": (True, True, True), # backed: every model passes + "93c0c0": (False, False, False), # claim 3 fits len 3, not the 2 bytes after the header + "91ddffffffff": (False, True, False), # running sum 1 + (2^32 - 1) wraps to 0 + "df80000000": (False, False, True), # map term 2 x 2^31 wraps to 0 in 32 bits + "ddffffffffdd00000001": (False, False, False), # the first term alone exceeds the budget +} + + +def walk_table() -> list[str | None]: + out: list[str | None] = [] + for hx, want in WALK_TABLE.items(): + w = dbr.walk(bytes.fromhex(hx)) + got = (w["nesting_depth"], w["declared_slots"], w["complete"], w["array_depth"]) + out.append(None if got == want else f"walk {hx}: {got} != {want}") + for hx, want in FLAG_TABLE.items(): + w = dbr.walk(bytes.fromhex(hx)) + got = (w["per_header_fits"], w["u32_add_fits"], w["u32_mul_fits"]) + out.append(None if got == want else f"walk flags {hx}: {got} != {want}") + out.append(expect_raises("walk 0xc1", lambda: dbr.walk(b"\xc1"), "never used")) + return out + + def cli_rejects(name: str, *args: str) -> str | None: """The CLI must exit non-zero on anything it does not understand (fail closed).""" rc = subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, check=False).returncode @@ -107,18 +158,30 @@ def main() -> None: del cut_accept["accept_vectors"][1]["reject_reasons"] results.append(expect_raises("accept complete", with_recipes(cut_accept), "not one complete document")) - # Coverage: drop every vector that separates each near-miss reader and watch its guard fire. - def without(test: Callable[[dict, dict], bool]) -> dict: + # Coverage: dropping exactly one discriminating vector must fire its guard. + def without(name: str) -> dict: doc = copy.deepcopy(good) - doc["reject_vectors"] = [v for v in doc["reject_vectors"] - if not test(v, dbr.walk(bytes.fromhex(v["input_hex"])))] + kept = [v for v in doc["reject_vectors"] if v["name"] != name] + if len(kept) != len(doc["reject_vectors"]) - 1: + sys.exit(f"FAIL coverage test names no vector: {name}") # a typo must not pass vacuously + doc["reject_vectors"] = kept return doc - results.append(expect_raises("coverage: sum", with_recipes(without( - lambda v, w: w["per_header_fits"] and v["reject_reasons"] == ["overclaim"])), "whole-document sum")) - results.append(expect_raises("coverage: u32", with_recipes(without( - lambda v, w: w["u32_sum_fits"])), "32-bit running sum")) - results.append(expect_raises("coverage: depth", with_recipes(without( - lambda v, w: v["nesting_depth"] == dbr.MAX_DEPTH_CEILING + 1)), "one level past the depth ceiling")) + for name, needle in (("nested_array16_each_header_fits_sum_overclaims", "per-header checks pass"), + ("array32_sum_wraps_u32_small_first", "32-bit running sum"), + ("map32_half_claim_wraps_u32_mul", "map term computed in 32 bits"), + ("nested_fixarray_depth_1025_complete", "complete array spine"), + ("nested_fixmap_depth_1025_complete", "complete map spine")): + results.append(expect_raises(f"coverage: drop {name}", with_recipes(without(name)), needle)) + + # Negative controls: a near-miss model forced to always pass must be caught. + real_walk = dbr.walk + for flag in ("per_header_fits", "u32_add_fits", "u32_mul_fits"): + def forced(data: bytes, flag: str = flag) -> dict: + return {**real_walk(data), flag: True} + with patch.object(dbr, "walk", forced): + results.append(expect_raises(f"control: {flag} always true", with_recipes(good), "the model is vacuous")) + + results.extend(walk_table()) results.append(expect_raises("require-extras", with_msgpack(None, good, require_extras=True), "not importable")) results.append(expect_raises("decoded reject", with_msgpack(lambda _: None, good), "decoded a reject vector")) From 420f5bacdec400517bb3ef423b1e27327e1317cf Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 27 Sep 2026 23:52:28 +1000 Subject: [PATCH 09/10] fix(interop): the guard-level MUST covers every read-path decode entry point (LAB-2503) A test that calls the structural guard directly passes even when the read path no longer calls it. The conformance MUST now requires driving each reject vector through every untrusted decode entry point on the SDK's read path (below its conversion to a cache miss) and asserting an error only the guard produces; a direct guard call alone is not enough. The CHANGELOG bullet mirrors it. Also: the TypeScript test cell says each vector trips the pre-scan or the event size cap ahead of it, and the "accept complete" mutation test finds its vector by name instead of by index. --- CHANGELOG.md | 4 +++- sdk-feature-matrix.md | 2 +- spec/interop-mode.md | 7 +++++-- tools/test_decode_bounds_reference.py | 5 +++-- 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ce3de31..1f035f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,9 @@ All notable changes to the CacheKit Protocol Specification. which derives every vector's depth and slot tags with a structural walk, and its mutation suite. The SDKs vendor earlier revisions; see the matrix's footnote 16. - An SDK's conformance test MUST assert that its structural guard rejects each reject - vector before materialising it. A verdict alone does not show when a reader rejected. + vector at every untrusted decode entry point on its read path, before materialising + it. A verdict alone does not show when a reader rejected, and a direct guard call + alone does not show that the read path runs the guard. - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) states the whole-document slot rule (per-header checks do not satisfy it; ext lengths count) for the envelope bytes and the payload inside them, and the diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index ae69915..3875efb 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -298,7 +298,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting each vector's pre-scan error | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting the guard error each vector trips (pre-scan, or the event size cap ahead of it) | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 87cb949..008342a 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -496,8 +496,11 @@ reader rejected: a stock decoder's default limits reject every reject vector tod and a reader with per-header checks alone rejects the incomplete ones at end of input, after it has pre-allocated for them. An SDK's conformance test MUST therefore assert that its structural guard rejects each reject vector before anything is materialised, -by calling the guard directly or by asserting the guard's distinguishing error. A run -that only asserts that a decode fails does not demonstrate conformance. +by driving each reject vector through every untrusted decode entry point on the SDK's +read path (below its conversion to a cache miss) and asserting an error that only the +guard produces; calling the guard directly as well is fine, but on its own does not +show that the read path runs it. A run that only asserts that a decode fails does not +demonstrate conformance. [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the bytes every decoder MUST reject (17) and MUST accept (3); the same rules apply to any other untrusted MessagePack decode in an SDK (auto-mode payloads after the diff --git a/tools/test_decode_bounds_reference.py b/tools/test_decode_bounds_reference.py index 5603c6f..d1d5790 100644 --- a/tools/test_decode_bounds_reference.py +++ b/tools/test_decode_bounds_reference.py @@ -153,9 +153,10 @@ def main() -> None: results.append(expect_raises("accept floor", with_recipes(deep_accept), "deeper than the floor")) cut_accept = copy.deepcopy(good) - cut_accept["accept_vectors"][1] = dbr.recipe("array16_256_backed_nils", "", "dc" + dbr.u16(256), 1, "c0" * 255, + at = next(i for i, v in enumerate(cut_accept["accept_vectors"]) if v["name"] == "array16_256_backed_nils") + cut_accept["accept_vectors"][at] = dbr.recipe("array16_256_backed_nils", "", "dc" + dbr.u16(256), 1, "c0" * 255, depth=1, slots=256, reasons=[]) - del cut_accept["accept_vectors"][1]["reject_reasons"] + del cut_accept["accept_vectors"][at]["reject_reasons"] results.append(expect_raises("accept complete", with_recipes(cut_accept), "not one complete document")) # Coverage: dropping exactly one discriminating vector must fire its guard. From 0ea6c5bad61c06d2deb52bf1266e28d76e192061 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 28 Sep 2026 00:00:09 +1000 Subject: [PATCH 10/10] docs(interop): conformance MUST names every untrusted entry point and admits a pre-decode size cap (LAB-2503) The conformance sentence covers value reads and any other untrusted decode such as invalidation events, measured below where the SDK turns the error into a cache miss or drops it, and accepts an error from any pre-decode check: the structural guard, or a size cap that entry point applies ahead of it. The CHANGELOG mirrors both bullets, and the TypeScript test cell is marked partial for its envelope entry point, which has no guard to assert. --- CHANGELOG.md | 12 +++++++----- sdk-feature-matrix.md | 2 +- spec/interop-mode.md | 12 +++++++----- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f035f8..58e21f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,10 +14,11 @@ All notable changes to the CacheKit Protocol Specification. (17 reject + 3 accept) with [`tools/decode-bounds-reference.py`](tools/decode-bounds-reference.py), which derives every vector's depth and slot tags with a structural walk, and its mutation suite. The SDKs vendor earlier revisions; see the matrix's footnote 16. -- An SDK's conformance test MUST assert that its structural guard rejects each reject - vector at every untrusted decode entry point on its read path, before materialising - it. A verdict alone does not show when a reader rejected, and a direct guard call - alone does not show that the read path runs the guard. +- An SDK's conformance test MUST assert that its structural guard rejects each + reject vector at every untrusted decode entry point (including invalidation + events), before materialising it; a size cap that rejects first also counts. A + verdict alone does not show when a reader rejected, and a direct guard call alone + does not show that the read path runs the guard. - [`spec/wire-format.md` → Security Limits](spec/wire-format.md#security-limits) states the whole-document slot rule (per-header checks do not satisfy it; ext lengths count) for the envelope bytes and the payload inside them, and the @@ -25,7 +26,8 @@ All notable changes to the CacheKit Protocol Specification. - Matrix: ByteStorage is ⚠️ in all three SDKs, because each decodes the envelope with `cachekit-core`'s `ByteStorage::retrieve`, which has no step-2 pre-scan. The decode-bounds test cells for Python and Rust are ⚠️ until their tests assert the - guard's rejection. + guard's rejection, and TypeScript's until its envelope entry point has a guard to + assert. - Open: the shared depth value, and any cap on the ~70× materialisation of *legal* payloads, stay [protocol#20](https://github.com/cachekit-io/protocol/issues/20)'s items. diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 3875efb..92dd170 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -298,7 +298,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting the guard error each vector trips (pre-scan, or the event size cap ahead of it) | ⚠️ Pending | +| Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack); `decode-bounds.json` vendored + CI-executed since [cachekit-py#276](https://github.com/cachekit-io/cachekit-py/pull/276) (LAB-2503) — ⚠️ asserts rejection and a peak-memory budget, not that the structural guard rejected (the [Decode bounds](spec/interop-mode.md#decode-bounds) MUST) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33); `decode-bounds.json` vendored + CI-executed since [cachekit-rs#73](https://github.com/cachekit-io/cachekit-rs/pull/73) (LAB-2503; default CI green on `main`) — ⚠️ asserts the error type only, not that the structural guard rejected | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors; decode bounds enforced ([#112](https://github.com/cachekit-io/cachekit-ts/pull/112)); `decode-bounds.json` vendored + CI-executed since [cachekit-ts#121](https://github.com/cachekit-io/cachekit-ts/pull/121) (LAB-2737), asserting the guard error each vector trips (pre-scan, or the event size cap ahead of it) — ⚠️ except the envelope entry point (`cachekit-core-ts` `unpack`), which has no guard to assert (see the Wire format row) | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | > [!NOTE] diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 008342a..b6e42a1 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -496,11 +496,13 @@ reader rejected: a stock decoder's default limits reject every reject vector tod and a reader with per-header checks alone rejects the incomplete ones at end of input, after it has pre-allocated for them. An SDK's conformance test MUST therefore assert that its structural guard rejects each reject vector before anything is materialised, -by driving each reject vector through every untrusted decode entry point on the SDK's -read path (below its conversion to a cache miss) and asserting an error that only the -guard produces; calling the guard directly as well is fine, but on its own does not -show that the read path runs it. A run that only asserts that a decode fails does not -demonstrate conformance. +by driving each reject vector through every untrusted decode entry point (value +reads, and any other untrusted decode such as invalidation events), below the point +where the SDK turns the error into a cache miss or drops it, and asserting an error +that only a pre-decode check produces: the structural guard, or a size cap that entry +point applies ahead of it. Calling the guard directly as well is fine, but on its own +does not show that the read path runs it. A run that only asserts that a decode fails +does not demonstrate conformance. [`test-vectors/decode-bounds.json`](../test-vectors/decode-bounds.json) pins the bytes every decoder MUST reject (17) and MUST accept (3); the same rules apply to any other untrusted MessagePack decode in an SDK (auto-mode payloads after the