From 8b00fd3179efb78b45869ddfdfcbf79052238557 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 28 Sep 2026 20:56:03 +1000 Subject: [PATCH 1/2] docs(wire-format): cachekit-core vendors fixture 1.1.1, so the bin16 coverage gap is closed (LAB-1750) The spec still described cachekit-core as vendoring fixture 1.1.0, with width_boundary_bin16_bin unchecked by the canonical writer. cachekit-core now pins 1.1.1 and re-encodes every bin twin, deriving the expected marker from the decoded compressed_data length. Replace the gap paragraph with the current state and the marker-derivation rule for anyone vendoring the fixture. --- CHANGELOG.md | 8 ++++++++ spec/wire-format.md | 44 +++++++++++++++++++++----------------------- 2 files changed, 29 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d9e8eb..e45332e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to the CacheKit Protocol Specification. ## [Unreleased] +### Wire format — vendored-fixture coverage note corrected (LAB-1750) + +- [`spec/wire-format.md`](spec/wire-format.md) no longer says `cachekit-core` vendors + fixture 1.1.0. It pins 1.1.1, so `width_boundary_bin16_bin` has a canonical-writer + (`lz4_flex`) compressed-byte and xxh3-64 checksum check. The section now states the rule + for anyone vendoring the fixture: derive each `*_bin` twin's expected marker from its + decoded `compressed_data` length, never assume bin8 or accept any `bin` width. + ### Encryption — default-tenant conformance vector (LAB-4666) - [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `default_tenant` diff --git a/spec/wire-format.md b/spec/wire-format.md index e0d3ae3..c8302be 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -41,8 +41,8 @@ This document specifies two layers: decode byte-identity for every vector and re-encode byte-identity for the canonical `*_bin` vectors only — legacy array-of-integers vectors are decode-only, retained as legacy-read proof. That re-encode assertion covers - only the vectors the pinned file contains (core currently vendors 1.1.0, with - the resulting gap detailed below). Byte-canonicity scopes to the + only the vectors the pinned file contains (core vendors 1.1.1, so every + `*_bin` twin, the `bin16` pair included). Byte-canonicity scopes to the envelope's MessagePack encoding and to the **canonical writer's** output: the LZ4 bytes inside `compressed_data` are not reproducible across conforming compressors — see @@ -287,31 +287,29 @@ bytes are therefore valid LZ4 block satisfies, so it accepts a re-pin to unrelated bytes. Neither half runs `lz4_flex`, so neither can detect an `lz4_flex` **behaviour** change; that remains the job of the re-encode assertions in `cachekit-core` described - below, subject to the vendored-version gap noted there. + below. This is the same doctrine [interop v2](interop-v2.md) records for its compressed-values profile. The pinned bytes are the **canonical implementation's** output (`lz4_flex` via `cachekit-core`), enforced by the re-encode byte-identity -assertions in `cachekit-core/tests/wire_format_vectors.rs` — **but only for the -vectors present in the fixture that repo vendors**. That matters today: -cachekit-core vendors 1.1.0 and pins `version == "1.1.0"`, so -`width_boundary_bin16` (added at 1.1.1) has **no canonical-writer (`lz4_flex`) -compressed-byte check anywhere in the fleet**, and its pinned xxh3-64 checksum -is recomputed nowhere. Its MessagePack encoding *is* covered: this repo's -`tools/wire-format-reference.py verify` asserts legacy and bin re-encode -byte-identity for it on every run, and liblz4 reproduces its compressed bytes -on the optional `lz4` leg — so do not read this gap as "the vector is -unverified". Closing it means re-vendoring 1.1.1 into cachekit-core, which -requires three changes together, not one: bump `FIXTURE_SHA256`, bump the -`version == "1.1.0"` pin to `1.1.1`, and relax -`assert_eq!(twin_bytes[1], 0xc4)` to accept `0xc5` — that assertion currently -requires *every* twin to be bin8, and `width_boundary_bin16_bin` is bin16 -(marker `0xc5`, 303-byte `compressed_data`), which is the whole point of the -vector. A drop-in re-vendor fails that test. The reference liblz4 mapping -above (`lz4.block`) is **decode-verified against every vector** in this repo's -CI (`tools/wire-format-reference.py verify`, optional `lz4` leg); on encode it -reproduces every pair except `large_compressible` byte-for-byte, which is an -observation, not a guarantee — but one this repo's CI pins (see +assertions in `cachekit-core/tests/wire_format_vectors.rs` — **for the vectors +present in the fixture that repo vendors**. cachekit-core vendors 1.1.1 and pins +`version == "1.1.1"`, so `width_boundary_bin16_bin` gets the canonical-writer +(`lz4_flex`) compressed-byte check and its xxh3-64 checksum is recomputed on +every run. That repo's bin-twin check derives each twin's expected marker from +the decoded `compressed_data` length (`≤255 → 0xc4`, `≤65535 → 0xc5`, else +`0xc6`) rather than accepting any `bin` width, so the `bin16` twin is held to +`0xc5` (303-byte `compressed_data`). Any consumer vendoring the fixture should +derive the marker the same way: an assertion that every twin is bin8 fails on +1.1.1, and one that accepts all three widths cannot detect a non-shortest +header. Independently, this repo's `tools/wire-format-reference.py verify` +asserts legacy and bin re-encode byte-identity for the pair on every run, and +liblz4 reproduces its compressed bytes on the optional `lz4` leg. The +reference liblz4 mapping above (`lz4.block`) is **decode-verified against +every vector** in this repo's CI (`tools/wire-format-reference.py verify`, +optional `lz4` leg); on encode it reproduces every pair except +`large_compressible` byte-for-byte, which is an observation, not a guarantee — +but one this repo's CI pins (see `LZ4_ENCODE_DIVERGENT`), so a toolchain change that alters the divergent set fails CI rather than quietly making this paragraph wrong. From 9d4b4bac20e2a1750a9360c8dd07b7dea04702d7 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Mon, 28 Sep 2026 21:03:36 +1000 Subject: [PATCH 2/2] docs(wire-format): state the marker rule once, without a dated vendored-version claim (LAB-1750) --- spec/wire-format.md | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/spec/wire-format.md b/spec/wire-format.md index c8302be..ba3b97e 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -41,8 +41,7 @@ This document specifies two layers: decode byte-identity for every vector and re-encode byte-identity for the canonical `*_bin` vectors only — legacy array-of-integers vectors are decode-only, retained as legacy-read proof. That re-encode assertion covers - only the vectors the pinned file contains (core vendors 1.1.1, so every - `*_bin` twin, the `bin16` pair included). Byte-canonicity scopes to the + only the vectors the pinned file contains. Byte-canonicity scopes to the envelope's MessagePack encoding and to the **canonical writer's** output: the LZ4 bytes inside `compressed_data` are not reproducible across conforming compressors — see @@ -293,19 +292,13 @@ This is the same doctrine [interop v2](interop-v2.md) records for its compressed-values profile. The pinned bytes are the **canonical implementation's** output (`lz4_flex` via `cachekit-core`), enforced by the re-encode byte-identity assertions in `cachekit-core/tests/wire_format_vectors.rs` — **for the vectors -present in the fixture that repo vendors**. cachekit-core vendors 1.1.1 and pins -`version == "1.1.1"`, so `width_boundary_bin16_bin` gets the canonical-writer -(`lz4_flex`) compressed-byte check and its xxh3-64 checksum is recomputed on -every run. That repo's bin-twin check derives each twin's expected marker from -the decoded `compressed_data` length (`≤255 → 0xc4`, `≤65535 → 0xc5`, else -`0xc6`) rather than accepting any `bin` width, so the `bin16` twin is held to -`0xc5` (303-byte `compressed_data`). Any consumer vendoring the fixture should -derive the marker the same way: an assertion that every twin is bin8 fails on -1.1.1, and one that accepts all three widths cannot detect a non-shortest -header. Independently, this repo's `tools/wire-format-reference.py verify` -asserts legacy and bin re-encode byte-identity for the pair on every run, and -liblz4 reproduces its compressed bytes on the optional `lz4` leg. The -reference liblz4 mapping above (`lz4.block`) is **decode-verified against +present in the fixture that repo vendors**, which recompute each twin's +`lz4_flex` bytes and xxh3-64 checksum. Anyone vendoring the fixture should +derive each `*_bin` twin's expected marker from its decoded `compressed_data` +length (`≤255 → 0xc4`, `≤65535 → 0xc5`, else `0xc6`), as cachekit-core does. An +assertion that every twin is bin8 fails on `width_boundary_bin16_bin` (`0xc5`, +303-byte `compressed_data`), and one that accepts all three widths cannot +detect a non-shortest header. The reference liblz4 mapping above (`lz4.block`) is **decode-verified against every vector** in this repo's CI (`tools/wire-format-reference.py verify`, optional `lz4` leg); on encode it reproduces every pair except `large_compressible` byte-for-byte, which is an observation, not a guarantee —