diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 597ef7cf2..5369a3385 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,54 @@ +## 2026-08-23 — the token-seam arc: four read-only research lanes, one probe, one vacuity audit + +- **Why:** an operator brief asked whether ONE versioned BPE tokenization can + simultaneously serve Tantivy lexical indexing, DeepNSM-v2 lexical/grammar + projection and an LSTM forward-prediction input surface, without + retokenizing, without rebuilding a DataFrame, and without a second cognitive + population — the integration half `E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1` (#1012) + explicitly did not ask. Tiering: Opus on the main thread for the architecture, + the probe and every gate; Sonnet for the bounded read-only lanes; no worker + ran cargo (the orchestrator compiled centrally, per the shared-target rule). +- **Lane A — DeepNSM-v2 lexical contract.** The decisive finding: the LIBRARY is + already tokenizer-free. `parse_to_spo(&[Tagged])` takes `(WordId, Pos)` and no + string; `split_whitespace`/`normalise` live only in the two examples. The seam + therefore needed no change to the crate. Also: `academic_20k.csv` is present + (20 845 rows, 18 559 distinct surface forms); `bible_vocab.txt` and the cam96 + codebook/codes are ABSENT. +- **Lane B — Polars / online-path falsifier.** Zero `polars` occurrences across + nine checkouts; every `DataFrame` mention is prose. `paperless-rs` and + `tesseract-rs` declare no arrow/datafusion/lance/lancedb. Also established + that `doc.v1` carries bbox/conf/leading_space and NO offset or span field — + which is the gap that blocks the seam on real scanned documents. +- **Lane C — Tantivy indexing-path audit.** The indexer never reads + `offset_from`/`offset_to` outside its own tests; snippets re-tokenize STORED + text at query time; `PreTokenizedString` costs ≈ `4 + 2N` allocations because + `segment_writer` deep-clones the boxed value. That last number is why the seam + uses a custom tokenizer with one reused `Token` buffer. +- **Lane D — SoA lane / continuation precedent.** No shipped token continuation + mechanism anywhere; the nearest in shape, `RailCarving::AxisSlab`, caps at 24 + levels — below the measured p50 of 4 particles. `ValueTenant` has 16 variants + and none for text. +- **The probe (main thread, Opus).** `PROBE-TOKEN-SEAM-1` — 37 gates, 13 + disable-runs verified red-then-green, in `AdaWorldAPI/paperless-rs` + (`crates/paperless-token`), with `docs/TOKEN-SEAM-ARCHITECTURE.md` as the + bounded architecture. Result: `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1`. +- **Lane E — vacuity audit (Sonnet, read-only, adversarial).** Found FIVE holes + in the finished probe, all real: a gate checking byte counts but not span + counts (the CRLF bug that collapsed 300 spans into 1 would have re-passed), a + threshold true by construction, an assertion about a type signature rather + than behaviour, an unconditional prefix check, and an unexercised + ASCII-vs-Unicode whitespace divergence. All five fixed; four gained their own + disable-runs; the fifth is bounded by a measured count of zero. +- **Two method failures worth the entry.** (1) Two disable-runs were themselves + wrong first — one relaxed a constant that does not bind on the fixture, one + targeted a mechanism the gate did not rest on — so both "passed" while proving + nothing. (2) An early disable batch reported "no failure" six times in a row + because the probe binary path was wrong and nothing ran at all. A null result + is a claim about the apparatus until proven otherwise. +- **Outcome:** #1017 (board), plus a paperless-rs commit that is **committed + locally and BLOCKED from pushing** — the GitHub App has no access to + `AdaWorldAPI/paperless-rs` for this org, verified through both the session + proxy and a proxy-bypassed attempt. ## 2026-08-23 — autoattended R2IL wave: 4 Sonnet probe workers + 1 Sonnet scribe + 1 Opus synthesis + 1 Haiku guarded executor - **Why:** operator directive to run the pattern autonomously — "sonnet agents diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index bd2f2302d..0350098a8 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,154 @@ +## 2026-08-23 — E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1 — the integration half of #1012: one versioned tokenization drives Tantivy, DeepNSM-v2 and a forward surface with zero re-tokenization, and the four gaps that stand between that and a carrier + +**Status:** FINDING — [MEASURED] (`PROBE-TOKEN-SEAM-1`, 37 gates, **13 +disable-runs each verified red-then-green**; two committed real corpora — the +in-tree KJV Genesis scene `PROBE-TOKEN-BPE-GEOMETRY-1` used, carried verbatim so +the numbers are comparable, and Project Gutenberg's *Alice* split into 300 +paragraphs, 75 514 B). The probe lives in `AdaWorldAPI/paperless-rs` +(`crates/paperless-token`, `docs/TOKEN-SEAM-ARCHITECTURE.md`) because it needs a +Tantivy dependency this workspace does not carry; this entry records the +**lance-graph-side** findings. +**This closes the integration half** of +`E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1` (#1012). That verdict measured that BPE FITS +the `6×(8:8)` geometry and refused to buy a carrier. It did not ask whether one +tokenization can SERVE several consumers at once. It can. +**Confidence:** High for what is measured at these two corpus scales; the 8-bit +lane saturated at 75 KB, so nothing here is a scale claim. + +**Headline.** ONE tokenization per span drove all three consumers and each added +**zero** further tokenizations — not by discipline but by construction. Totals: +313 source tokenizations for 308 spans plus 5 deliberate fixtures, and 1 QUERY +tokenization on a deliberately separate counter (a query is different bytes; +folding it into one number would make the claim a lie). + +**The two facts that make it adoptable, and neither was designed for this:** + +1. **DeepNSM-v2's library is already tokenizer-free.** `parse_to_spo(&[Tagged])` + consumes `(WordId, Pos)` pairs and touches no string; the + `split_whitespace`/`normalise` logic lives ONLY in `examples/bible_wave.rs` + and `examples/genre_shapes.rs`. The seam needed **no change to the crate**. + The `(WordId, Pos)` boundary is the shipped seam and nobody had used it as + one. +2. **Tantivy structurally cannot own offsets.** Its indexer reads `Token::text` + and `Token::position`, uses `position_length` transiently, and reads + `offset_from`/`offset_to` NOWHERE outside its own tests. Offsets are consumed + only by snippet generation, which re-tokenizes the STORED text at query time. + An index cannot become the ABI here even by accident. + +**Measured, and the numbers are the point rather than the verdict:** + +| | KJV scene | Alice | +|---|---|---| +| bytes / spans / tokens | 1 126 / 8 / 354 | 75 514 / 300 / 37 149 | +| compression | 3.18× | **2.03×** | +| distinct ids used | 137 | **247 of 255** | +| resident lane bytes | 832 (74 % of source) | 55 572 (74 %) | +| receipt share of resident | 54 % | 30 % | +| particles/span p50/p95/max | 4 / 8 / 8 | 8 / 30 / 43 | +| tokens per lexical unit p50/max | 1 / 7 | 2 / 15 | +| tokens straddling a word boundary | 30 | 58 | + +- **The 8-bit lane SATURATES.** 247 of 255 ids on 75 KB of ordinary English, with + compression already fallen from 3.18× to 2.03×. The canon's answer — the hi + byte of each `(8:8)` pair as a PAGE lane, two separate bytes, never a widened + `u16` — is **untested**. Until it is measured no scale claim about token BPE + should be made, and this supersedes any reading of #1012's 3.35× as a + corpus-independent figure. +- **The resident lane is ~74 % of the source text, not a fraction of it**, and + at these span sizes **framing is 30–54 % of it**. A 56-byte receipt against + 12-byte particles means the RECEIPT's column layout matters more than the + particle's. #1012 could not see this — it had no receipt. +- **Cardinality is not 1:1 in either direction**, so a BPE↔`WordId` projection + is a real function, not a relabelling. Nothing in the seam assigns a `WordId` + to a BPE token; that would be a second vocabulary wearing DeepNSM's + coordinate system. +- **Byte offsets are DERIVED**, by prefix sum over a per-id decoded-length + table. The receipt stores no offset column at all. + +**Four lance-graph-side gaps, each named rather than worked around:** + +1. **No shipped token continuation mechanism anywhere.** The nearest precedent + in SHAPE is `rail_geometry::RailCarving::AxisSlab { reg, cont: Option }`, + which chains one register to one continuation and caps at `RAIL_MAX_DEPTH = 24` + levels — **below the measured p50 of 4 particles**, so it does not fit. The + probe uses a contiguous run (`first_particle + particle_count + token_count`). + Stated honestly there are TWO lawful framings and the trade is exact: + `particle_count` alone bounds the run and a PAD scan inside that bound is + already exact BECAUSE PAD is reserved (cost: one vocabulary slot, which at a + 255-cap that saturates is not free); or `token_count` costs 4 bytes and frees + the slot for a full 256-id alphabet. What is unlawful is inferring the end + from padding with no bound — measured, a lane-wide PAD scan overshoots + receipt 0 by 10 tokens straight into receipt 1. +2. **`ValueTenant` has no token variant** (16 discriminants, none for text), so a + lawful resident lane must implement `SoaEnvelope` or land as a new tenant. + The probe's lane is a probe-local `Vec` and says so. +3. **There is no callable part-of-speech surface, and the reason is a decision + already taken.** `coca_pos`/`archaic_pos`/`normalise` are byte-identical in + BOTH deepnsm-v2 examples, above a comment stating `deepnsm_v2::lexicon` was + DELETED after an audit found `lance-graph-planner`'s `insight_coca_read` + already grounds it. That grounding does not reach a lean consumer: + `insight_coca_read` is itself an **example binary**, in a crate carrying + `serde`/`serde_yml`/`tokio`/`ndarray`, and its master `lexicon.tsv` is absent + from this checkout. The probe restated the twenty-line tagger rather than + re-litigate the deletion — recorded so the next consumer has the evidence the + audit did not. +4. **The semantic half is unexercised.** `cam96_codebook.bin` / `cam96_codes.bin` + are release assets, absent here, so palette256² DISTANCE never ran. Only the + lexical/grammar half was measured. + +**Polars: refuted, and the honest form is weaker than the question invited.** A +sweep of nine checkouts found **zero** `polars` occurrences in any manifest or +source; every `DataFrame` mention is prose. `paperless-rs` and `tesseract-rs` +declare none of arrow/datafusion/lance/lancedb. There was nothing to remove. The +structured-evidence path is likewise not tabular algebra: +`lance-graph-arm-discovery` takes `Dataset { spec: FeatureSpec, rows: +Vec> }` — category-index rows against a schema. + +**Method note, and it is the transferable part.** An independent vacuity audit of +the finished probe found FIVE holes: a gate asserting byte counts but never span +counts (the CRLF bug that collapsed 300 spans into 1 would have re-passed it), a +threshold true by construction, an assertion about a type signature rather than +behaviour, an unconditional prefix check, and an unexercised ASCII-vs-Unicode +whitespace divergence. All five are fixed; four gained their own disable-runs; +the fifth is bounded by a measured count of 0. Separately, TWO disable-runs were +themselves wrong first — one relaxed a knob that does not bind on the fixture, +one targeted a mechanism the gate did not actually rest on — and an early batch +reported "no failure" six times in a row because the probe binary path was wrong +and nothing ran. **A knob that does not bind is not a disable; a fixture's SHAPE +is part of a test's coverage; and a null result is a claim about the apparatus +until proven otherwise.** + +**⚠ SELF-CORRECTION, same session, after being pointed at `ogar-doc-ir`.** Two +claims above were wrong and are corrected here rather than left standing: + +1. **The seam invented an identity that already existed.** The first cut minted + `source_id`/`span_id` integers. `ogar_doc_ir::DocIr` already answers all + three questions a tokenization receipt asks — `content_sha256` for WHICH + document, `(DocPage::number, Region::reading_order)` for WHICH span, and + `Region::text` for the span's canonical text. The probe was re-cut to read + them (`docir.rs`; gates `T-DOCIR` / `T-DOCIR-KEY` / `T-DOCIR-SPANS`, 41 + total, 18 disable-runs). Note the crate's own docs CORRECT its plan's first + sketch on what that hash is: a **per-acquisition dedup key**, not a + cross-retina identity — which is exactly the right reading for a receipt, + because you tokenize bytes. +2. **"The OCR boundary supplies no byte offsets" is RETIRED as a gap.** It + supplies no PAGE-wide offset and does not need to: a region owns its text, + so an offset is region-local, and `ogar-from-docv1::region_text` is where the + `leading_space`-aware join already happens. What remains is far smaller — a + sub-region span needs a non-zero `byte_from`, which the receipt already + carries and no producer emits. + +Also corrected: the `247 of 255` figure quoted above is the count of ids +APPEARING in the lane, not the vocabulary size. Measured, the trained table is +**full at 255/255** on Alice (and on the whole 170 KB file), and **180 of 255** +on the KJV fixture — where, as #1016's own record of that fixture says, the +CORPUS rather than the cap set it. The saturation conclusion holds and is +stronger; the number was the wrong quantity. + +**Untouched by this.** HHTL is address geometry and BPE is tokenization — +#1012's measured refutation of the merge tree as a radix prefix partition +stands. Content never travels in classid: the contract id is a FIELD on the +receipt, gated by a grep of the library's own non-comment source. ## 2026-08-23 — E-THE-SEVEN-OPCODE-PROJECTION-IS-NOT-X86-AND-THE-CHAIN-CARRIER-WINS-1 — four wave probes: the chain carrier confirmed, the vocabulary survives optimization, and the boundary that qualifies all of it **Status:** FINDING — [MEASURED] × 4 (`PROBE-R2IL-OPTIMIZATION-TRANSFER-1` 5/5, diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index b44ee386a..28a39109e 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,32 @@ +## 2026-08-23 — #1017 OPEN — the token seam: #1012's integration half answered, and the 8-bit lane's ceiling found + +- **What exists now:** `PROBE-TOKEN-SEAM-1` (37 gates, 13 disable-runs) in + `AdaWorldAPI/paperless-rs crates/paperless-token`, with + `docs/TOKEN-SEAM-ARCHITECTURE.md` as its bounded architecture. It answers the + question #1012 left open: ONE versioned BPE tokenization of a span drives + Tantivy, DeepNSM-v2 and a forward-prediction input surface simultaneously, + each consumer BORROWING, none re-tokenizing. Neither Tantivy nor DeepNSM-v2 + needed a line changed. +- **Standing laws banked:** ONE SOURCE SPAN → ONE TOKENIZATION RECEIPT; + TOKENIZE ONCE, PROJECT MANY TIMES; AN INDEX MAY ACCELERATE THE ABI, IT MUST + NEVER BECOME THE ABI; BPE SEQUENCE IDENTITY IS NOT A SEMANTIC WORD + COORDINATE; and (method) A KNOB THAT DOES NOT BIND IS NOT A DISABLE — a + fixture's SHAPE is part of a test's coverage. +- **The number that bounds prior work:** the 8-bit id lane saturates at 75 KB + (247/255 ids, compression 3.18× → 2.03×). #1012's 3.35× is a 1 KB figure and + must not be read as corpus-independent. The hi-byte PAGE lane is the next + probe and no scale claim survives without it. +- **Open, in order:** (1) the paged vocabulary; (2) a real retina — this probe + builds its `DocIr` from text, so the next one should take + `ogar-from-docv1` on an actual scan and a `spider_doc_ir` crawl of the same + content and check both present the same span-population shape; + (3) a lawful resident lane (`SoaEnvelope` or a new `ValueTenant`, designed + against the measured 30–54 % framing overhead); (4) a real forward arm — + the seam supplies the input, but which representation a trained model prefers + is untested; (5) a structured-evidence corpus to exercise the parallel typed + path (`arm-discovery`'s `FeatureSpec` + category-index rows) and the shared + span identity where the two meet. + ## 2026-08-23 — #1006..#1014 MERGED — the belief-ABI arc: Step 1 audit, root-law probes, Step 2 ruling request, frontier Phases 1+2, the real-episode measurement Nine PRs (#1010/#1011 landing stacked via #1009's merge) closed the arc: diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 76cabc148..a4f2e2991 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,42 @@ +## 2026-08-23 — lance-graph #1017 (OPEN) — the integration half of #1012: one receipt, three borrowed consumers + +- **Added:** `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1` — the board record of + `PROBE-TOKEN-SEAM-1` (37 gates, 13 disable-runs red-then-green; probe code in + `AdaWorldAPI/paperless-rs crates/paperless-token` + `docs/TOKEN-SEAM-ARCHITECTURE.md`, + which lives there because it needs a Tantivy dep this workspace does not carry). + ONE tokenization per span drove Tantivy, DeepNSM-v2 and a forward-prediction + surface; each added ZERO further tokenizations (313 source for 308 spans + 5 + fixtures; 1 query on a separate counter). +- **Locked:** DeepNSM-v2's library is ALREADY the seam — `parse_to_spo(&[Tagged])` + takes `(WordId, Pos)` and no string, so the crate needed no change; a BPE token + is never assigned a `WordId` (different id spaces, cardinality measured + non-1:1 in both directions); byte offsets are DERIVED by prefix sum over a + per-id length table, so a receipt stores no offset column; Tantivy cannot own + offsets (its indexer never reads them). +- **Measured, and it bounds #1012's headline:** the 8-bit lane SATURATES — 247 + of 255 ids on 75 KB, compression 3.18× → **2.03×**. The resident lane is 74 % + of source and framing is 30–54 % of THAT (56-byte receipt vs 12-byte + particles), so the receipt's layout outranks the particle's. +- **Self-corrected in-session:** the first cut minted `source_id`/`span_id`; + re-cut onto `ogar_doc_ir::DocIr` (`content_sha256` + `(page, reading_order)` + + `Region::text`), so the receipt mints nothing. That RETIRES the + "no byte offsets at the OCR boundary" gap — offsets are region-local — and + corrects `247 of 255` (ids appearing in the lane) to a table that is FULL at + 255/255 on Alice and 180/255 on the KJV fixture. +- **Deferred / named:** no shipped token continuation mechanism (the + `RailCarving::AxisSlab` precedent caps at 24 levels, under the measured p50 of + 4 particles); `ValueTenant` has no token variant; no callable PoS surface — + `deepnsm_v2::lexicon` was deliberately deleted and the `insight_coca_read` + grounding cited for it is an example binary outside a lean consumer's + dependency barrier; cam96 codebook/codes ABSENT so the semantic half never + ran; the hi-byte PAGE lane untested, which is the next probe. +- **Refuted:** Polars in the online path — zero occurrences across nine + checkouts; `paperless-rs`/`tesseract-rs` declare no arrow/datafusion/lance/lancedb + at all. There was nothing to remove. +- **Docs:** `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1`; LATEST_STATE updated. +- **Confidence:** High for the two measured corpora; no scale claim — the + vocabulary was full at 75 KB. + ## 2026-08-23 — GAP FILLED — arc rows for #976..#1005 (consolidated by the orchestrator from a wave scribe's primary-source reconstruction) The gap marker recorded below this block is now DISCHARGED for #977..#1005. diff --git a/.claude/harvest/paperless-token-plateau/0001-token-seam.patch b/.claude/harvest/paperless-token-plateau/0001-token-seam.patch new file mode 100644 index 000000000..630c270fa --- /dev/null +++ b/.claude/harvest/paperless-token-plateau/0001-token-seam.patch @@ -0,0 +1,5511 @@ +From b0e5e78307373503373c61c20046a900b272c1c0 Mon Sep 17 00:00:00 2001 +From: Claude +Date: Sun, 23 Aug 2026 23:04:46 +0000 +Subject: [PATCH 1/4] =?UTF-8?q?one=20tokenization=20receipt,=20three=20bor?= + =?UTF-8?q?rowed=20consumers=20=E2=80=94=20the=20seam=20probe?= +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +`PROBE-TOKEN-SEAM-1` answers the integration half of the question #1012's +CAN-FIT-NOT-YET-BUY verdict left open: can ONE versioned BPE tokenization of a +source span drive Tantivy, DeepNSM-v2 and a forward-prediction surface at once, +with no re-tokenization, no DataFrame, and no second population? + +Measured yes, on two committed real corpora (the in-tree KJV scene #1012 used, +carried verbatim so the numbers are comparable, and Alice split into 300 +paragraphs). 37 gates, 13 disable-runs each verified red-then-green. + + - 313 source tokenizations for 308 spans plus 5 deliberate fixtures — exactly + one per span; the three consumers add ZERO. Query analysis is counted on a + separate counter because a query is different bytes. + - Tantivy needs no patch: the indexed field value is a receipt HANDLE, so the + index is never handed the source at all. Positions in the index ARE the + receipt's positions. Measured in this fork, the indexer never reads + offset_from/offset_to — the index structurally cannot own offsets. + - DeepNSM-v2 needs no change: its library is already tokenizer-free + (`parse_to_spo(&[Tagged])`), and `project()` takes a borrowed view and NO + source bytes, so re-reading the source is unavailable rather than avoided. + - Byte offsets are DERIVED from a per-id length table; the receipt stores no + offset column. + - Cardinality is not 1:1 in either direction: a lexical unit spans a median + of 2 tokens (max 15) and 58 tokens straddle a word boundary. + +What the probe does NOT settle is nine named gaps, two of them load-bearing: +the OCR boundary hands over no byte offsets to attach a span to, and the 8-bit +vocabulary lane saturated at 75 KB (247 of 255 ids), where compression had +already fallen from 3.18x to 2.03x. + +Polars is not removed from the online path because it was never in it: zero +occurrences across nine checkouts, and neither paperless-rs nor tesseract-rs +declares arrow/datafusion/lance/lancedb at all. + +An independent vacuity audit of the finished probe found five holes — a gate +that checked byte counts but not span counts (the CRLF bug would have +re-passed), a threshold true by construction, an assertion about a type +signature rather than behaviour, an unconditional prefix check, and an +unexercised ASCII-vs-Unicode whitespace divergence. All five are fixed and the +first four now have their own disable-runs. + +The workspace lance-graph pin moves bfc4f6d -> 51c54fa as one deliberate act: +the seam consumes deepnsm-v2 from the same repo, and two revs of one git URL +would give two copies of lance-graph-contract whose types do not unify. + +Co-Authored-By: Claude Opus 5 +Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf +--- + CLAUDE.md | 14 +- + Cargo.toml | 21 +- + crates/paperless-token/Cargo.toml | 30 + + crates/paperless-token/README.md | 78 ++ + .../examples/probe_token_seam.rs | 1091 +++++++++++++++++ + crates/paperless-token/src/contract.rs | 332 +++++ + crates/paperless-token/src/forward.rs | 130 ++ + crates/paperless-token/src/lane.rs | 260 ++++ + crates/paperless-token/src/lexical.rs | 128 ++ + crates/paperless-token/src/lib.rs | 39 + + crates/paperless-token/src/seam_tantivy.rs | 204 +++ + docs/TOKEN-SEAM-ARCHITECTURE.md | 500 ++++++++ + 12 files changed, 2821 insertions(+), 6 deletions(-) + create mode 100644 crates/paperless-token/Cargo.toml + create mode 100644 crates/paperless-token/README.md + create mode 100644 crates/paperless-token/examples/probe_token_seam.rs + create mode 100644 crates/paperless-token/src/contract.rs + create mode 100644 crates/paperless-token/src/forward.rs + create mode 100644 crates/paperless-token/src/lane.rs + create mode 100644 crates/paperless-token/src/lexical.rs + create mode 100644 crates/paperless-token/src/lib.rs + create mode 100644 crates/paperless-token/src/seam_tantivy.rs + create mode 100644 docs/TOKEN-SEAM-ARCHITECTURE.md + +diff --git a/CLAUDE.md b/CLAUDE.md +index 007f9f9..80c9a99 100644 +--- a/CLAUDE.md ++++ b/CLAUDE.md +@@ -102,7 +102,11 @@ auto-track `stable`; bump explicitly with the gates green. + ``` + crates/ + ├── paperless-kv/ the KV layout — S-2 preflight gate, document subtree keys +-└── paperless-intake/ raw bytes → gate → tesseract-rs recognition → doc.v1 ++├── paperless-intake/ raw bytes → gate → tesseract-rs recognition → doc.v1 ++└── paperless-token/ the tokenization seam — ONE versioned BPE receipt, ++ borrowed by Tantivy, DeepNSM-v2 and a forward surface ++docs/ ++└── TOKEN-SEAM-ARCHITECTURE.md the bounded architecture + the measured probe + ``` + + ## Status — honest +@@ -113,6 +117,14 @@ crates/ + `surreal_container::open` returns `Err(Blocked)` and every module is a + `// TODO task NN` header). That is also what `W4-8` prescribes: *"No storage + backend chosen (KV blob is the consumer's)."* ++- `paperless-token`: a **probe**, and it is honest about being one — 37 gates, ++ 9 disable-runs verified red-then-green, two committed real corpora. It proves ++ the seam (one tokenization per span; Tantivy, DeepNSM-v2 and a forward ++ surface all borrowed off it; zero changes needed to either consumer crate) ++ and names eight gaps that stand between it and a production carrier. The ++ resident lane is still a probe-local `Vec<[u8;12]>`, not a lawful ++ `SoaEnvelope` lane. Read `docs/TOKEN-SEAM-ARCHITECTURE.md` before extending ++ it; §7 is the list of what is NOT settled. + - `paperless-intake`: **a stub.** It exists so the `[patch]` chain is actually + exercised by the crate graph rather than declared and unused. Recognition is + not wired yet. +diff --git a/Cargo.toml b/Cargo.toml +index 6e5d5ee..50f16e6 100644 +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -1,6 +1,6 @@ + [workspace] + resolver = "2" +-members = ["crates/paperless-kv", "crates/paperless-intake"] ++members = ["crates/paperless-kv", "crates/paperless-intake", "crates/paperless-token"] + + [workspace.package] + version = "0.0.1" +@@ -23,7 +23,11 @@ repository = "https://github.com/AdaWorldAPI/paperless-rs" + # forbidden: a floating ref makes a build unreproducible and silently changes + # what a measurement measured. + # +-# lance-graph @ bfc4f6d ++# lance-graph @ 51c54fa (bumped 2026-08-23 from bfc4f6d: the token seam ++# consumes `deepnsm-v2`, which lives in the SAME repo. ++# Two revs of one git URL would give two copies of ++# `lance-graph-contract` whose types do not unify, so ++# the pin moves as ONE deliberate act, not per-crate.) + # OGAR @ 70296be (contains the merged ingestion spine, PR #266) + # tesseract-rs @ b369a97 + # ndarray @ 52dc61e +@@ -52,13 +56,20 @@ repository = "https://github.com/AdaWorldAPI/paperless-rs" + [workspace.dependencies] + tesseract-ogar = { git = "https://github.com/AdaWorldAPI/tesseract-rs", rev = "b369a9758d642d1b7f3a02b3aca2f53497f74531" } + ogar-vocab = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a5db99dd3fa92a532ba060eb0838" } +-lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "bfc4f6db700b17943967a86f196cbd040f57a333" } ++lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } ++# The semantic consumer of the token seam. Standalone crate (its own empty ++# `[workspace]` table) inside the same checkout; only dep is the contract crate ++# above, resolved by a path that stays INSIDE the checkout, so it needs no patch. ++deepnsm-v2 = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } ++# The lexical index. Fork per P0; default features off (in-RAM probe index, and ++# the default set drags zstd-sys, a C build). ++tantivy = { git = "https://github.com/AdaWorldAPI/tantivy", rev = "5fc9fccb613c9838cf7fde77f73dc4afa3216375", default-features = false } + sha2 = { version = "0.10", default-features = false } + + [patch."https://github.com/AdaWorldAPI/tesseract-rs"] + ogar-vocab = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a5db99dd3fa92a532ba060eb0838" } +-lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "bfc4f6db700b17943967a86f196cbd040f57a333" } +-deepnsm = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "bfc4f6db700b17943967a86f196cbd040f57a333" } ++lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } ++deepnsm = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } + ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", rev = "52dc61ed2284ab2ce2fdde5143e5d3cd618aefaf" } + + [patch."https://github.com/AdaWorldAPI/lance-graph"] +diff --git a/crates/paperless-token/Cargo.toml b/crates/paperless-token/Cargo.toml +new file mode 100644 +index 0000000..1bc624a +--- /dev/null ++++ b/crates/paperless-token/Cargo.toml +@@ -0,0 +1,30 @@ ++[package] ++name = "paperless-token" ++description = "The tokenization seam: one versioned BPE receipt, borrowed by Tantivy, DeepNSM-v2 and a forward-prediction surface. Probe, not a production carrier." ++version.workspace = true ++edition.workspace = true ++rust-version.workspace = true ++license.workspace = true ++repository.workspace = true ++ ++[dependencies] ++# The contract identity. Already a workspace dep for the S-2 convergence key. ++sha2.workspace = true ++ ++# The lexical index. AdaWorldAPI fork, fixed rev, default features OFF: the ++# probe indexes in RAM, so mmap/lz4/zstd/stemmer are all dead weight here — and ++# `columnar-zstd-compression` would drag `zstd-sys`, a C build, into a tree ++# whose whole proposition is that it has no C in it. ++tantivy = { workspace = true } ++ ++# The semantic consumer. Standalone crate inside the lance-graph checkout; its ++# only dependency is the zero-dep contract crate, so this edge is light. ++deepnsm-v2 = { workspace = true } ++ ++[lints.rust] ++unsafe_code = "forbid" ++missing_docs = "warn" ++ ++[lints.clippy] ++all = { level = "deny", priority = -1 } ++pedantic = { level = "warn", priority = -1 } +diff --git a/crates/paperless-token/README.md b/crates/paperless-token/README.md +new file mode 100644 +index 0000000..9e93c84 +--- /dev/null ++++ b/crates/paperless-token/README.md +@@ -0,0 +1,78 @@ ++# paperless-token ++ ++A bounded PROBE of the tokenization seam — not a production carrier. It asks ++whether a single versioned BPE tokenization of one source span can drive ++lexical retrieval (Tantivy), lexical/grammar projection (DeepNSM-v2), and a ++forward-prediction input surface at the same time, without re-tokenizing, ++without a DataFrame, and without a second cognitive population. The law it ++tests: **ONE TOKENIZATION RECEIPT, MANY BORROWED CONSUMERS.** ++ ++## Run it ++ ++```sh ++cargo run --release -p paperless-token --example probe_token_seam ++``` ++ ++`--release` is not optional in spirit: the encode cost is reported as ++merge-table probe COUNTS, and a debug run would still be correct but slow — ++Alice trains a 255-cap merge table over 75 KB of paragraph text. ++ ++## Fixture provenance ++ ++| file | bytes | sha256 | source | licence/derivation | ++|---|---|---|---|---| ++| `corpus/alice.txt` | 174,693 | `15124d40c182677c2d90fba80310173d63428e0591ce0df3e9bdc01a789a89c6` | carried from `/home/user/tantivy/benches/alice.txt` | Project Gutenberg, *Alice's Adventures in Wonderland*, public domain. CRLF line terminators, UTF-8 with BOM — the probe strips both before splitting into paragraphs. | ++| `corpus/coca_academic_20k.tsv` | 226,651 | `4ae20ce39dd3018346700e0f88df2b59e1a7df4e4a06e0f285fd44065166e0f0` | derived from `/home/user/lance-graph/crates/deepnsm/word_frequency/academic_20k.csv` (sha256 `1dfd5edaa5a6ac9b8ac5abbf87894abaf7de8a449ad7c09ca1f6324226396e2d`, 20,845 data rows + 1 header) | `awk -F, 'NR>1 {print $4"\t"$5}' academic_20k.csv > coca_academic_20k.tsv` — columns 4 and 5 of that CSV are `word` and `Pos`. | ++ ++### Why the fixtures are committed and not fetched ++ ++A test fixture that lives outside the repo is a time bomb with a skip-guard ++for a fuse — on a fresh container the skip makes it pass instead of failing ++honestly. This repo's own `CLAUDE.md` states the rule plainly: "Fixtures live ++in the repo." Both files above are committed; nothing in this crate reaches ++outside the checkout at build or run time. ++ ++### The one mirror, and its drift check ++ ++`corpus/coca_academic_20k.tsv` is a DERIVED copy of data that lives ++authoritatively in `lance-graph`. It is the exact `word,Pos` projection ++DeepNSM-v2's own `genre_shapes` example loads, carried here so this crate ++stays hermetic. To confirm it has not drifted from its source, given a ++`lance-graph` checkout at ``: ++ ++```sh ++awk -F, 'NR>1 {print $4"\t"$5}' /crates/deepnsm/word_frequency/academic_20k.csv \ ++ | diff - corpus/coca_academic_20k.tsv && echo IN-SYNC ++``` ++ ++## What is ABSENT ++ ++- **The whole-KJV corpus.** Only the Genesis 2-3 scene (`SCENE` in the probe) ++ is carried, verbatim from `PROBE-TOKEN-BPE-GEOMETRY-1`, so the two probes ++ measure the same bytes. No larger KJV text is fetched, synthesized, or ++ simulated. ++- **`bible_vocab.txt`.** Not present on this machine; not reconstructed. ++- **The trained `cam96_codebook.bin` / `cam96_codes.bin`.** Their absence ++ means DeepNSM's semantic DISTANCE half cannot be exercised by this probe — ++ only the lexical/grammar half (tagging, PoS, SPO extraction) runs. ++- **Any MecCog corpus.** It does not exist anywhere on this machine in any ++ form. The only trace of it anywhere in this workspace is one rhetorical ++ mention in a MedCare-rs board file — not a corpus, not a fixture, not code. ++ ++Each of these is reported as absent where the probe's output would otherwise ++imply it, and none is stood in for with a simulated or synthetic substitute. ++ ++## Layering ++ ++| module | owns | never | ++|---|---|---| ++| `contract` | the codebook + its identity | reads the lane | ++| `lane` | resident particles + framing | owns text | ++| `lexical` | the DeepNSM projection | reads the source | ++| `seam_tantivy` | the index seam | owns offsets | ++| `forward` | the prediction input surface | owns the sequence | ++ ++The probe's own gate list (`PROBE-TOKEN-SEAM-1`, run via the command above) ++exists to PROVE each "never" — e.g. `T-CONTRACT-GATE` checks that a view ++refuses to open under the wrong contract — rather than merely asserting the ++boundary in prose. +diff --git a/crates/paperless-token/examples/probe_token_seam.rs b/crates/paperless-token/examples/probe_token_seam.rs +new file mode 100644 +index 0000000..a72b778 +--- /dev/null ++++ b/crates/paperless-token/examples/probe_token_seam.rs +@@ -0,0 +1,1091 @@ ++//! `PROBE-TOKEN-SEAM-1` — one versioned tokenization receipt, three borrowed ++//! consumers: Tantivy, DeepNSM-v2, and a forward-prediction input surface. ++//! ++//! ```text ++//! ONE SOURCE SPAN -> ONE TOKENIZATION RECEIPT. ++//! TOKENIZE ONCE. PROJECT MANY TIMES. ++//! AN INDEX MAY ACCELERATE THE ABI. IT MUST NEVER BECOME THE ABI. ++//! ``` ++//! ++//! # Honesty box ++//! ++//! - **Corpora are real and committed.** `corpus/alice.txt` is Project ++//! Gutenberg's *Alice's Adventures in Wonderland*, carried from ++//! `tantivy/benches/alice.txt` so this probe is hermetic (a fixture that ++//! lives outside the repo is a time bomb with a skip-guard for a fuse). ++//! `corpus/coca_academic_20k.tsv` is the `word,Pos` projection of ++//! `lance-graph/crates/deepnsm/word_frequency/academic_20k.csv` ++//! (sha256 `1dfd5eda…`, 20 845 data rows), the exact file DeepNSM-v2's own ++//! `genre_shapes` example loads. The KJV scene is the real in-tree text ++//! `PROBE-TOKEN-BPE-GEOMETRY-1` used, carried verbatim so the two probes are ++//! comparable. ++//! - **ABSENT, reported not simulated:** the whole-KJV corpus, `bible_vocab.txt`, ++//! the trained `cam96_codebook.bin` / `cam96_codes.bin` (so `DeepNSM`'s semantic ++//! DISTANCE cannot be exercised here — only the lexical/grammar half), and any ++//! `MecCog` corpus, which does not exist on this machine in any form. ++//! - **No wall-time claims.** Cost is reported as operation counts and bytes. ++//! - **The forward arm is a counting baseline, not a language model.** It exists ++//! to prove the INPUT SURFACE, and its accuracy number is labelled as such. ++ ++// A probe reads like a lab notebook: `g` is the gate, `c` the corpus, `v` the ++// view, `s` the summary. Renaming them to `gate`/`corpus`/`view`/`summary` ++// would make every assertion line wrap and read worse, and the scope of each ++// is a dozen lines. Same for the two long `main`/`tantivy_arm` bodies: a probe ++// is one linear argument and splitting it into helpers hides the order the ++// gates run in, which is the thing a reader checks. ++#![allow( ++ clippy::many_single_char_names, ++ clippy::too_many_lines, ++ clippy::cast_precision_loss ++)] ++ ++use std::collections::HashMap; ++use std::sync::Arc; ++ ++use deepnsm_v2::{parse_to_spo, PaletteVocab, Pos, Spo, Tagged}; ++use paperless_token::contract::{query_passes, source_passes, NormRule, TokenizerContract}; ++use paperless_token::forward::{score, windows, CountPredictor}; ++use paperless_token::lane::{TokenLane, TokenStreamReceipt, IDS_PER_PARTICLE}; ++use paperless_token::lexical::project; ++use paperless_token::seam_tantivy::{handle, ReceiptTokenizer, SeamStore, TermMode}; ++use tantivy::collector::TopDocs; ++use tantivy::query::PhraseQuery; ++use tantivy::schema::{IndexRecordOption, Schema, TextFieldIndexing, TextOptions, Value}; ++use tantivy::tokenizer::TokenStream as _; ++use tantivy::tokenizer::Tokenizer as _; ++use tantivy::{Directory, Index, IndexWriter, TantivyDocument, Term}; ++ ++/// The real KJV Genesis 2-3 scene, carried verbatim from ++/// `PROBE-TOKEN-BPE-GEOMETRY-1` so the two probes measure the same bytes. ++const SCENE: &[&str] = &[ ++ "But of the tree of the knowledge of good and evil, thou shalt not eat of it: for in \ ++ the day that thou eatest thereof thou shalt surely die.", ++ "And they were both naked, the man and his wife, and were not ashamed.", ++ "Now the serpent was more subtil than any beast of the field which the LORD God had \ ++ made. And he said unto the woman, Yea, hath God said, Ye shall not eat of every tree \ ++ of the garden?", ++ "And the serpent said unto the woman, Ye shall not surely die:", ++ "And when the woman saw that the tree was good for food, and that it was pleasant to \ ++ the eyes, and a tree to be desired to make one wise, she took of the fruit thereof, \ ++ and did eat, and gave also unto her husband with her; and he did eat.", ++ "And the eyes of them both were opened, and they knew that they were naked; and they \ ++ sewed fig leaves together, and made themselves aprons.", ++ "And they heard the voice of the LORD God walking in the garden in the cool of the \ ++ day: and Adam and his wife hid themselves from the presence of the LORD God amongst \ ++ the trees of the garden.", ++ "And he said, I heard thy voice in the garden, and I was afraid, because I was naked; \ ++ and I hid myself.", ++]; ++ ++/// One corpus under test. ++struct Corpus { ++ name: &'static str, ++ /// The canonical text, and the span boundaries within it. ++ text: String, ++ spans: Vec<(u32, u32)>, ++} ++ ++fn kjv() -> Corpus { ++ let mut text = String::new(); ++ let mut spans = Vec::new(); ++ for v in SCENE { ++ let from = u32::try_from(text.len()).expect("fits"); ++ text.push_str(v); ++ text.push('\n'); ++ spans.push((from, u32::try_from(text.len()).expect("fits"))); ++ } ++ Corpus { ++ name: "kjv-genesis-scene", ++ text, ++ spans, ++ } ++} ++ ++/// Alice, split into paragraphs. Blank-line separated; a paragraph is the span. ++fn alice(max_spans: usize) -> Corpus { ++ // The committed file is CRLF with a BOM. A naive `split("\n\n")` finds ++ // NOTHING in it — the first version of this probe silently produced ONE ++ // 169 KB span and every per-span number it printed was meaningless while ++ // every gate stayed green. Normalising first is what makes the span count ++ // real. ++ let raw = include_str!("../corpus/alice.txt") ++ .trim_start_matches('\u{feff}') ++ .replace("\r\n", "\n"); ++ let mut text = String::new(); ++ let mut spans = Vec::new(); ++ for para in raw.split("\n\n") { ++ let p = para.split_whitespace().collect::>().join(" "); ++ if p.len() < 40 { ++ continue; ++ } ++ let from = u32::try_from(text.len()).expect("fits"); ++ text.push_str(&p); ++ text.push('\n'); ++ spans.push((from, u32::try_from(text.len()).expect("fits"))); ++ if spans.len() >= max_spans { ++ break; ++ } ++ } ++ Corpus { ++ name: "alice-paragraphs", ++ text, ++ spans, ++ } ++} ++ ++/// `word -> Pos`, from the committed COCA projection. The COCA letter mapping is ++/// `DeepNSM`'s own; it lives in BOTH of that crate's examples, byte-identical, ++/// because `deepnsm_v2::lexicon` was DELETED by an earlier audit on the ground ++/// that `lance-graph-planner`'s `insight_coca_read` already grounds it. That ++/// grounding does not reach here: `insight_coca_read` is itself an example ++/// binary in a crate outside this repo's dependency barrier. So re-stating the ++/// twenty-line tagger is the ONE duplication this seam forces, and the report ++/// records the deletion rather than quietly re-adding the module. ++fn coca_pos(letter: &str) -> Pos { ++ match letter { ++ "n" | "p" => Pos::Noun, ++ "v" => Pos::Verb, ++ "j" => Pos::Adj, ++ "a" | "d" => Pos::Det, ++ _ => Pos::Other, ++ } ++} ++ ++fn load_vocab() -> (PaletteVocab, HashMap) { ++ let tsv = include_str!("../corpus/coca_academic_20k.tsv"); ++ let mut vocab = PaletteVocab::new(); ++ let mut pos = HashMap::new(); ++ let mut words: Vec<&str> = Vec::new(); ++ for line in tsv.lines() { ++ let mut it = line.split('\t'); ++ let (Some(w), Some(p)) = (it.next(), it.next()) else { ++ continue; ++ }; ++ if w.is_empty() { ++ continue; ++ } ++ words.push(w); ++ pos.entry(w.to_string()).or_insert_with(|| coca_pos(p)); ++ } ++ vocab.from_frequency_ranked(words); ++ (vocab, pos) ++} ++ ++fn pct(a: usize, b: usize) -> f64 { ++ if b == 0 { ++ 0.0 ++ } else { ++ 100.0 * a as f64 / b as f64 ++ } ++} ++ ++fn quantiles(v: &mut [usize]) -> (usize, usize, usize) { ++ v.sort_unstable(); ++ if v.is_empty() { ++ return (0, 0, 0); ++ } ++ ( ++ v[v.len() / 2], ++ v[(v.len() * 95) / 100], ++ *v.last().expect("non-empty"), ++ ) ++} ++ ++struct Gate { ++ pass: u32, ++} ++ ++impl Gate { ++ fn run(&mut self, name: &str, ok: bool, detail: &str) { ++ assert!(ok, "[FAIL] {name} — {detail}"); ++ println!(" [PASS] {name} — {detail}"); ++ self.pass += 1; ++ } ++} ++ ++/// What one corpus measured. ++struct Summary { ++ name: &'static str, ++ bytes: usize, ++ spans: usize, ++ tokens: usize, ++ uniq_tokens: usize, ++ particles: usize, ++ resident_bytes: usize, ++ p50: usize, ++ p95: usize, ++ pmax: usize, ++ continuation_rate: f64, ++ encode_probes: usize, ++ lex_units: usize, ++ lex_resolved: usize, ++ tokens_per_unit: (usize, usize, usize), ++ multi_unit_tokens: usize, ++ spo: usize, ++} ++ ++/// Tokenize a whole corpus into ONE lane under ONE contract, then measure it. ++fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, Summary) { ++ let mut lane = TokenLane::new(); ++ let mut probes = 0usize; ++ let mut per_span: Vec = Vec::new(); ++ let before = source_passes(); ++ for (i, &(from, to)) in c.spans.iter().enumerate() { ++ let bytes = &c.text.as_bytes()[from as usize..to as usize]; ++ let (tokens, p) = contract ++ .try_encode(bytes) ++ .expect("contract trained on this corpus"); ++ probes += p; ++ per_span.push(tokens.len().div_ceil(IDS_PER_PARTICLE)); ++ lane.append(0, u32::try_from(i).expect("fits"), from, contract, &tokens); ++ } ++ let passes = source_passes() - before; ++ g.run( ++ &format!( ++ "T-PASSES[{}] exactly one source tokenization per span", ++ c.name ++ ), ++ passes == c.spans.len(), ++ &format!( ++ "{passes} source tokenizations for {} spans (1.000 per span); the three consumers \ ++ below add ZERO further source passes — every one of them reads the lane", ++ c.spans.len() ++ ), ++ ); ++ ++ // ---- reconstruction + framing ---- ++ let mut recon_ok = true; ++ let mut tokens_total = 0usize; ++ let mut uniq = std::collections::HashSet::new(); ++ for (r, &(from, to)) in lane.receipts().iter().zip(&c.spans) { ++ let v = lane.view(r, contract).expect("same contract"); ++ tokens_total += v.len(); ++ uniq.extend(v.ids().iter().copied()); ++ if v.decode() != c.text.as_bytes()[from as usize..to as usize] { ++ recon_ok = false; ++ } ++ } ++ let (p50, p95, pmax) = quantiles(&mut per_span.clone()); ++ let cont = per_span.iter().filter(|&&n| n > 1).count(); ++ g.run( ++ &format!( ++ "T-RECON[{}] every span decodes byte-exact from ids alone", ++ c.name ++ ), ++ recon_ok, ++ &format!( ++ "{} spans, {tokens_total} tokens, {} distinct ids; decode reads the lane and the \ ++ codebook and nothing else — the canonical text stays authoritative and is never \ ++ consulted to read a span back", ++ c.spans.len(), ++ uniq.len() ++ ), ++ ); ++ ++ // ---- derived offsets ---- ++ let mut off_ok = true; ++ for (r, &(from, _)) in lane.receipts().iter().zip(&c.spans) { ++ let v = lane.view(r, contract).expect("same contract"); ++ // Ground truth computed the expensive way: decode each prefix. ++ let mut cursor = from; ++ for t in v.tokens() { ++ let truth_len = contract.decode(&[t.id]).0.len(); ++ if t.byte_from != cursor ++ || t.byte_to != cursor + u32::try_from(truth_len).expect("short") ++ { ++ off_ok = false; ++ } ++ cursor = t.byte_to; ++ } ++ if cursor != c.spans[r.span_id as usize].1 { ++ off_ok = false; ++ } ++ } ++ g.run( ++ &format!( ++ "T-OFFSET[{}] byte offsets are DERIVED, never stored", ++ c.name ++ ), ++ off_ok, ++ &format!( ++ "every token's (byte_from, byte_to) reproduces a decode-the-prefix ground truth, \ ++ and the last token's end lands exactly on the span end; the receipt carries no \ ++ offset column — {} bytes/receipt total, of which zero are offsets per token", ++ core::mem::size_of::() ++ ), ++ ); ++ ++ let summary = Summary { ++ name: c.name, ++ bytes: c.text.len(), ++ spans: c.spans.len(), ++ tokens: tokens_total, ++ uniq_tokens: uniq.len(), ++ particles: lane.particle_len(), ++ resident_bytes: lane.resident_bytes(), ++ p50, ++ p95, ++ pmax, ++ continuation_rate: pct(cont, per_span.len()), ++ encode_probes: probes, ++ lex_units: 0, ++ lex_resolved: 0, ++ tokens_per_unit: (0, 0, 0), ++ multi_unit_tokens: 0, ++ spo: 0, ++ }; ++ (lane, summary) ++} ++ ++/// Consumer B: the DeepNSM-v2 projection, run from ids alone. ++fn deepnsm_arm( ++ c: &Corpus, ++ contract: &TokenizerContract, ++ lane: &TokenLane, ++ vocab: &PaletteVocab, ++ posmap: &HashMap, ++ s: &mut Summary, ++ g: &mut Gate, ++) { ++ let before = source_passes(); ++ let mut units = 0usize; ++ let mut resolved = 0usize; ++ let mut spans_per_unit: Vec = Vec::new(); ++ let mut multi_unit_tokens = 0usize; ++ let mut triples: Vec = Vec::new(); ++ let mut flattened: Vec = Vec::new(); ++ let mut offsets_ok = true; ++ ++ for r in lane.receipts() { ++ let v = lane.view(r, contract).expect("same contract"); ++ let lex = project(&v); ++ // A token that carries more than one unit's start is a token straddling ++ // a word boundary — the cardinality that refutes any 1:1 assumption. ++ let mut starts: HashMap = HashMap::new(); ++ let mut tagged: Vec = Vec::new(); ++ for u in &lex { ++ units += 1; ++ spans_per_unit.push(u.token_span as usize); ++ *starts.entry(u.first_token).or_insert(0) += 1; ++ // The unit's byte span must address the canonical text and land on ++ // exactly the bytes it claims (modulo the normalisation that drops ++ // non-alphabetic characters inside a word). ++ let raw = &c.text.as_bytes()[u.byte_from as usize..u.byte_to as usize]; ++ let renorm: String = raw ++ .iter() ++ .filter(|b| b.is_ascii_alphabetic()) ++ .map(|b| b.to_ascii_lowercase() as char) ++ .collect(); ++ if renorm != u.surface { ++ offsets_ok = false; ++ } ++ if let Some(id) = vocab.id(&u.surface) { ++ resolved += 1; ++ let pos = posmap.get(&u.surface).copied().unwrap_or(Pos::Other); ++ tagged.push(Tagged::new(id, pos)); ++ } ++ } ++ multi_unit_tokens += starts.values().filter(|&&n| n > 1).count(); ++ tagged.push(Tagged::new(0, Pos::Stop)); ++ triples.extend(parse_to_spo(&tagged)); ++ // Control: the SAME word ids with every tag flattened to Noun. If the ++ // FSM's output did not depend on the Pos half of the pair, this would ++ // produce the same triples — and "the FSM consumes (WordId, Pos)" ++ // would be a claim about a type signature rather than about behaviour. ++ let blind: Vec = tagged ++ .iter() ++ .map(|t| { ++ Tagged::new( ++ t.id, ++ if t.pos == Pos::Stop { ++ Pos::Stop ++ } else { ++ Pos::Noun ++ }, ++ ) ++ }) ++ .collect(); ++ flattened.extend(parse_to_spo(&blind)); ++ } ++ let passes = source_passes() - before; ++ let (q50, q95, qmax) = quantiles(&mut spans_per_unit.clone()); ++ ++ g.run( ++ &format!( ++ "T-DEEPNSM[{}] projection runs from ids alone, zero source passes", ++ c.name ++ ), ++ passes == 0 && units > 0, ++ &format!( ++ "{units} lexical units from {} tokens with {passes} additional source \ ++ tokenizations — `project()` takes a borrowed view and NO source bytes, so \ ++ re-reading the source is unavailable rather than merely avoided", ++ s.tokens ++ ), ++ ); ++ g.run( ++ &format!( ++ "T-DEEPNSM-SPAN[{}] every unit's byte span addresses the canonical text", ++ c.name ++ ), ++ offsets_ok, ++ "each unit's (byte_from, byte_to) slice of the canonical text re-normalises to exactly \ ++ the surface the projection produced from ids — the seam's span identity is real, not \ ++ nominal", ++ ); ++ g.run( ++ &format!( ++ "T-DEEPNSM-CARD[{}] the BPE:word cardinality is measured, not assumed", ++ c.name ++ ), ++ qmax > 1 && multi_unit_tokens > 0 && q50 >= 1, ++ &format!( ++ "tokens per lexical unit p50={q50} p95={q95} max={qmax}; {multi_unit_tokens} tokens \ ++ carry the start of MORE THAN ONE unit (a token straddling a word boundary). \ ++ Neither direction is 1:1 — BPE sequence identity and the DeepNSM word coordinate \ ++ are different id spaces, exactly as the fence requires" ++ ), ++ ); ++ g.run( ++ &format!( ++ "T-DEEPNSM-FSM[{}] the FSM consumes (WordId, Pos) with no strings", ++ c.name ++ ), ++ !triples.is_empty() && flattened.is_empty() && resolved * 2 > units, ++ &format!( ++ "{}/{units} units resolved to a PaletteVocab WordId ({:.1}% ; {:.1}% OOV against a \ ++ {}-word academic vocabulary), and `parse_to_spo` emitted {} SPO triples. The Pos \ ++ half is load-bearing, not decorative: the SAME word ids with every tag flattened to \ ++ Noun emit {} triples, so the FSM's output genuinely depends on the pair and this \ ++ gate is not a restatement of a type signature. DeepNSM-v2 needed NO change — its \ ++ library is already tokenizer-free, the split_whitespace lives only in its examples", ++ resolved, ++ pct(resolved, units), ++ 100.0 - pct(resolved, units), ++ vocab.len(), ++ triples.len(), ++ flattened.len() ++ ), ++ ); ++ s.lex_units = units; ++ s.lex_resolved = resolved; ++ s.tokens_per_unit = (q50, q95, qmax); ++ s.multi_unit_tokens = multi_unit_tokens; ++ s.spo = triples.len(); ++} ++ ++/// What one Tantivy index measured. ++struct IndexStats { ++ segments: usize, ++ terms: usize, ++ docs: u64, ++ total_tokens: u64, ++ bytes: usize, ++ phrase_hits: usize, ++} ++ ++/// Consumer A: Tantivy, driven by receipt handles. The index never sees text. ++fn tantivy_arm( ++ store: &Arc, ++ mode: TermMode, ++ probe_receipt: usize, ++ g: &mut Gate, ++ label: &str, ++) -> IndexStats { ++ let mut sb = Schema::builder(); ++ let opts = TextOptions::default().set_stored().set_indexing_options( ++ TextFieldIndexing::default() ++ .set_tokenizer("receipt") ++ .set_index_option(IndexRecordOption::WithFreqsAndPositions), ++ ); ++ let body = sb.add_text_field("body", opts); ++ let schema = sb.build(); ++ let index = Index::create_in_ram(schema); ++ index ++ .tokenizers() ++ .register("receipt", ReceiptTokenizer::new(Arc::clone(store), mode)); ++ ++ let n = store.lane.receipts().len(); ++ let before_src = source_passes(); ++ let before_q = query_passes(); ++ { ++ // ONE indexing thread => ONE segment. The first version used the ++ // default writer, which spreads documents across threads and therefore ++ // across segments; summing `num_terms()` over segments then reported ++ // 202 terms for a lane holding 137 distinct ids. A term count that ++ // exceeds the vocabulary is the signature of that mistake. ++ let mut w: IndexWriter = index ++ .writer_with_num_threads(1, 50_000_000) ++ .expect("writer"); ++ for i in 0..n { ++ let mut d = TantivyDocument::default(); ++ // The FIELD VALUE IS A HANDLE, not the text. This is what makes ++ // re-tokenization structurally impossible rather than merely ++ // discouraged. ++ d.add_text(body, handle(i)); ++ w.add_document(d).expect("add"); ++ } ++ w.commit().expect("commit"); ++ } ++ let src_passes = source_passes() - before_src; ++ let q_passes = query_passes() - before_q; ++ ++ let reader = index.reader().expect("reader"); ++ let searcher = reader.searcher(); ++ let mut terms = 0usize; ++ let mut total_tokens = 0u64; ++ let segments = searcher.segment_readers().len(); ++ for seg in searcher.segment_readers() { ++ let inv = seg.inverted_index(body).expect("inverted index"); ++ terms += inv.terms().num_terms(); ++ total_tokens += inv.total_num_tokens(); ++ } ++ let dir = index.directory(); ++ let bytes: usize = dir ++ .list_managed_files() ++ .iter() ++ .filter_map(|p| dir.open_read(p).ok()) ++ .map(|f| { ++ use tantivy::HasLen as _; ++ f.len() ++ }) ++ .sum(); ++ ++ // ---- positions must BE the receipt's positions ---- ++ let mut analyzer = index.tokenizers().get("receipt").expect("registered"); ++ let h = handle(probe_receipt); ++ let mut seen: Vec = Vec::new(); ++ { ++ let mut ts = analyzer.token_stream(&h); ++ while ts.advance() { ++ seen.push(ts.token().position); ++ } ++ } ++ let r = store.lane.receipts()[probe_receipt]; ++ let expect: Vec = (0..r.token_count as usize).collect(); ++ g.run( ++ &format!("T-TANTIVY-POS[{label}] index positions ARE receipt positions"), ++ seen == expect, ++ &format!( ++ "receipt {probe_receipt} has token_count={} and the analyzer registered on the index \ ++ yields positions 0..{} in order — no drift, because there is only one segmentation \ ++ and the index consumed it", ++ r.token_count, r.token_count ++ ), ++ ); ++ g.run( ++ &format!("T-TANTIVY-NOSRC[{label}] indexing tokenized NOTHING, by either counter"), ++ src_passes == 0 && q_passes == 0, ++ &format!( ++ "{n} documents indexed, {src_passes} source + {q_passes} query tokenizations. Both \ ++ counters are asserted: during INDEXING any tokenization at all is a \ ++ re-tokenization, and handing Tantivy the raw text instead of a handle would take \ ++ the fallback path and bump the QUERY counter — a source-only assertion would have \ ++ stayed green through exactly the mistake it exists to catch" ++ ), ++ ); ++ ++ // ---- a phrase query over the SAME segmentation ---- ++ let ids: Vec = { ++ let v = store.lane.view(&r, &store.contract).expect("same contract"); ++ v.ids().iter().copied().take(3).collect() ++ }; ++ let phrase: Vec = ids ++ .iter() ++ .map(|&id| { ++ let text = match mode { ++ TermMode::Surface => { ++ String::from_utf8_lossy(store.contract.surface(id)).to_string() ++ } ++ TermMode::TokenId => format!("{id:02x}"), ++ }; ++ Term::from_field_text(body, &text) ++ }) ++ .collect(); ++ let (hits, stored) = if phrase.len() >= 2 { ++ let q = PhraseQuery::new(phrase); ++ let top = searcher ++ .search(&q, &TopDocs::with_limit(10).order_by_score()) ++ .expect("search"); ++ let stored = top.first().map_or_else(String::new, |(_, addr)| { ++ let d: TantivyDocument = searcher.doc(*addr).expect("doc"); ++ d.get_first(body) ++ .and_then(|v| v.as_str().map(str::to_string)) ++ .unwrap_or_default() ++ }); ++ (top.len(), stored) ++ } else { ++ (0, String::new()) ++ }; ++ g.run( ++ &format!("T-TANTIVY-PHRASE[{label}] a phrase over the shared segmentation retrieves"), ++ hits > 0 && stored == handle(probe_receipt), ++ &format!( ++ "a 3-token phrase taken straight out of receipt {probe_receipt}'s ids matched {hits} \ ++ document(s) via POSITIONS ONLY, and the top hit's stored field is exactly \ ++ \"{stored}\" — the receipt the phrase came FROM, not merely some handle-shaped \ ++ value (every document in this index starts with the handle prefix, so a \ ++ prefix check would have been unconditional). Tantivy persists term + position and never writes \ ++ offset_from/offset_to at all (measured in this fork: the indexer's `index_text` \ ++ reads `text` and `position`, uses `position_length` transiently, and reads the \ ++ offsets nowhere outside its own tests), so the index cannot become the owner of \ ++ offsets even by accident" ++ ), ++ ); ++ ++ IndexStats { ++ segments, ++ terms, ++ docs: searcher.num_docs(), ++ total_tokens, ++ bytes, ++ phrase_hits: hits, ++ } ++} ++ ++/// Lines of Rust that are not comments — the fence must not be satisfied by ++/// its own prose. ++fn code_lines(src: &str) -> String { ++ src.lines() ++ .filter(|l| { ++ let t = l.trim_start(); ++ !(t.starts_with("//") || t.starts_with("/*") || t.starts_with('*')) ++ }) ++ .collect::>() ++ .join("\n") ++} ++ ++fn main() { ++ let mut g = Gate { pass: 0 }; ++ println!("PROBE-TOKEN-SEAM-1 — one receipt, three borrowed consumers\n"); ++ ++ let (vocab, posmap) = load_vocab(); ++ let corpora = [kjv(), alice(300)]; ++ ++ // ---- T-CORPUS ---- ++ // The unicode-whitespace divergence, BOUNDED by measurement rather than by ++ // hope: DeepNSM splits on `char::is_whitespace` (Unicode), this projection ++ // splits on `u8::is_ascii_whitespace`. They differ only where a corpus ++ // contains non-ASCII whitespace, so the probe counts it instead of ++ // assuming it away. ++ let exotic_ws: usize = corpora ++ .iter() ++ .map(|c| { ++ c.text ++ .chars() ++ .filter(|ch| ch.is_whitespace() && !ch.is_ascii_whitespace()) ++ .count() ++ }) ++ .sum(); ++ g.run( ++ "T-CORPUS committed, real, hermetic; span population and the unicode-whitespace \ ++ divergence both measured", ++ corpora.iter().all(|c| c.text.len() > 1000) ++ && corpora[0].spans.len() >= 8 ++ && corpora[1].spans.len() >= 100 ++ && vocab.len() > 10_000 ++ && exotic_ws == 0, ++ &format!( ++ "{} ({} bytes, {} spans) and {} ({} bytes, {} spans), both committed in-repo; \ ++ DeepNSM vocabulary {} words from the committed COCA projection. ABSENT and NOT \ ++ simulated: whole-KJV, bible_vocab.txt, the trained cam96 codebook/codes (so the \ ++ semantic DISTANCE half of DeepNSM cannot be exercised here), and any MecCog corpus \ ++ — which exists nowhere on this machine. The span COUNT is asserted, not just \ ++ the byte count: the CRLF split bug this probe hit left every byte total healthy \ ++ while collapsing 300 spans into 1. Non-ASCII whitespace, on which this \ ++ projection's byte-level split would legitimately diverge from DeepNSM's \ ++ char-level `is_whitespace`, occurs {exotic_ws} times across both corpora — the \ ++ divergence is bounded by measurement, and named as a gap rather than denied", ++ corpora[0].name, ++ corpora[0].text.len(), ++ corpora[0].spans.len(), ++ corpora[1].name, ++ corpora[1].text.len(), ++ corpora[1].spans.len(), ++ vocab.len() ++ ), ++ ); ++ ++ let mut summaries: Vec = Vec::new(); ++ let mut index_rows: Vec<(String, IndexStats)> = Vec::new(); ++ let mut forward_rows: Vec<(String, usize, f64, usize, f64)> = Vec::new(); ++ ++ for c in &corpora { ++ let contract = TokenizerContract::train(c.text.as_bytes(), NormRule::Identity); ++ let (lane, mut s) = ingest(c, &contract, &mut g); ++ deepnsm_arm(c, &contract, &lane, &vocab, &posmap, &mut s, &mut g); ++ ++ // ---- Consumer C: the forward-prediction input surface ---- ++ let before = source_passes(); ++ let split = lane.receipts().len() * 4 / 5; ++ let holdout: Vec<_> = lane.receipts()[split..] ++ .iter() ++ .filter_map(|r| lane.view(r, &contract)) ++ .collect(); ++ let mut best = (0usize, -1.0f64); ++ let mut base = 0.0f64; ++ let mut by_k: Vec<(usize, f64)> = Vec::new(); ++ let mut best_sc = paperless_token::forward::ForwardScore::default(); ++ for k in [1usize, 2, 3] { ++ let mut p = CountPredictor::new(k); ++ for r in &lane.receipts()[..split] { ++ if let Some(v) = lane.view(r, &contract) { ++ p.observe(&v); ++ } ++ } ++ let sc = score(&p, &holdout); ++ by_k.push((k, sc.accuracy())); ++ if k == 1 { ++ base = sc.accuracy(); ++ } ++ if sc.accuracy() > best.1 { ++ best = (k, sc.accuracy()); ++ best_sc = sc; ++ } ++ } ++ let fwd_passes = source_passes() - before; ++ // How many token positions have a DeepNSM coordinate available? ++ let mut covered = 0usize; ++ let mut positions = 0usize; ++ for r in lane.receipts() { ++ let v = lane.view(r, &contract).expect("same contract"); ++ positions += v.len(); ++ for u in project(&v) { ++ if vocab.id(&u.surface).is_some() { ++ covered += u.token_span as usize; ++ } ++ } ++ } ++ let cov = pct(covered, positions); ++ let borrowed = { ++ let r = lane.receipts()[0]; ++ let v = lane.view(&r, &contract).expect("same contract"); ++ let ids_ptr = v.ids().as_ptr(); ++ let lane_ptr = lane.particles().as_ptr().cast::(); ++ // The window really is a slice of the resident population, not a copy. ++ ids_ptr == lane_ptr && windows(&v, 2).count() == v.len().saturating_sub(2) ++ }; ++ g.run( ++ &format!( ++ "T-FORWARD[{}] the input surface is BORROWED, and it is the same ids", ++ s.name ++ ), ++ borrowed && fwd_passes == 0, ++ &format!( ++ "context windows are slices INTO the resident particle array (pointer-identical \ ++ to the lane's own storage), {fwd_passes} extra source tokenizations; an order-k \ ++ counting BASELINE — not a language model — reaches top-1 {:.1}% at k={} against \ ++ {:.1}% at k=1 on held-out spans ({} positions scored, {} with a context never \ ++ seen in training; per order {}), and {cov:.1}% of token positions also carry a \ ++ DeepNSM (basin, identity) coordinate, so the hybrid representation is \ ++ CONSTRUCTIBLE. Which representation a trained model should prefer is NOT \ ++ measured here and no claim is made", ++ 100.0 * best.1, ++ best.0, ++ 100.0 * base, ++ best_sc.scored, ++ best_sc.unseen, ++ by_k.iter() ++ .map(|(k, a)| format!("k={k}:{:.1}%", 100.0 * a)) ++ .collect::>() ++ .join(" ") ++ ), ++ ); ++ forward_rows.push((s.name.to_string(), best.0, best.1, positions, cov)); ++ ++ // ---- Consumer A: Tantivy, both term modes ---- ++ let store = Arc::new(SeamStore { ++ contract: contract.clone(), ++ lane: lane.clone(), ++ }); ++ for (mode, label) in [ ++ (TermMode::Surface, "surface"), ++ (TermMode::TokenId, "token-id"), ++ ] { ++ let st = tantivy_arm(&store, mode, 0, &mut g, &format!("{}/{label}", s.name)); ++ index_rows.push((format!("{}/{label}", s.name), st)); ++ } ++ summaries.push(s); ++ } ++ ++ // ---- T-CONTRACT: the codebook law ---- ++ let c0 = &corpora[0]; ++ let a = TokenizerContract::train(c0.text.as_bytes(), NormRule::Identity); ++ let b = TokenizerContract::train(c0.text.as_bytes(), NormRule::Identity); ++ let other = TokenizerContract::train(corpora[1].text.as_bytes(), NormRule::Identity); ++ let lower = TokenizerContract::train(c0.text.as_bytes(), NormRule::AsciiLowercase); ++ let sample = c0.text.as_bytes(); ++ let (ta, _) = a.try_encode(sample).expect("trained on it"); ++ let (tb, _) = b.try_encode(sample).expect("trained on it"); ++ g.run( ++ "T-CONTRACT-DET same codebook + same source -> byte-identical token stream", ++ a.contract_id() == b.contract_id() && ta == tb, ++ &format!( ++ "two independent trainings of the same corpus produce contract id {}… and identical \ ++ {}-token streams; the tie-break is deterministic, so the identity is a function of \ ++ the corpus, not of allocation order", ++ &a.contract_hex()[..16], ++ ta.len() ++ ), ++ ); ++ let cross = other.decode(&ta).0; ++ g.run( ++ "T-CONTRACT-DIFF a changed codebook is a DIFFERENT contract, and ids alone are meaningless", ++ a.contract_id() != other.contract_id() ++ && a.contract_id() != lower.contract_id() ++ && cross != sample, ++ &format!( ++ "a different corpus yields contract {}… and a different NORMALISATION RULE over the \ ++ SAME corpus yields {}… — the rule is hashed into the identity, not left implicit. \ ++ Decoding corpus-A's ids under corpus-B's codebook returns {} bytes of garbage \ ++ instead of the {} original bytes: a stored id without its contract is not a weak \ ++ reference, it is a wrong one", ++ &other.contract_hex()[..16], ++ &lower.contract_hex()[..16], ++ cross.len(), ++ sample.len() ++ ), ++ ); ++ ++ // ---- T-CONTRACT-RULE: the normalisation rule is IN the identity ---- ++ // Trained on already-lowercase bytes, `Identity` and `AsciiLowercase` ++ // produce the SAME table — so if the rule were not hashed, they would share ++ // an id while behaving differently on mixed-case input. That is the only ++ // corpus shape on which this claim is falsifiable; a mixed-case corpus ++ // makes the tables differ and the assertion passes for the wrong reason. ++ let lc = c0.text.to_ascii_lowercase(); ++ let r_id = TokenizerContract::train(lc.as_bytes(), NormRule::Identity); ++ let r_lo = TokenizerContract::train(lc.as_bytes(), NormRule::AsciiLowercase); ++ let mixed = b"The Garden"; ++ let id_takes_mixed = r_id.try_encode(mixed).is_some(); ++ let lo_takes_mixed = r_lo.try_encode(mixed).is_some(); ++ g.run( ++ "T-CONTRACT-RULE identical tables + different rules are different contracts", ++ r_id.vocab_len() == r_lo.vocab_len() ++ && r_id.contract_id() != r_lo.contract_id() ++ && !id_takes_mixed ++ && lo_takes_mixed, ++ &format!( ++ "on an already-lowercase corpus both rules train the SAME {}-id table, yet the \ ++ contract ids differ ({}… vs {}…) because the rule is hashed into the identity. \ ++ They are not interchangeable: encoding \"The Garden\" under Identity is REFUSED \ ++ (uppercase 'T' is outside the trained alphabet) and under AsciiLowercase succeeds. \ ++ A shared id for those two behaviours would be a lie the store could not detect", ++ r_id.vocab_len(), ++ &r_id.contract_hex()[..12], ++ &r_lo.contract_hex()[..12] ++ ), ++ ); ++ ++ // ---- T-FRAME: token_count is authoritative, PAD is not a length ---- ++ let mut frame_lane = TokenLane::new(); ++ let (full, _) = a.try_encode(c0.text.as_bytes()).expect("trained"); ++ let exact = full.len() - (full.len() % IDS_PER_PARTICLE); // a 12-aligned run ++ let r0 = frame_lane.append(0, 0, 0, &a, &full[..exact]); ++ let r1 = frame_lane.append(0, 1, 0, &a, &full[exact..]); ++ let flat = frame_lane.particles().as_flattened(); ++ let pad_scan = flat ++ .iter() ++ .position(|&x| x == paperless_token::contract::PAD) ++ .unwrap_or(flat.len()); ++ g.run( ++ "T-FRAME an exactly-full run has NO pad, so pad-inference reads into the next receipt", ++ r0.tail_is_full() && pad_scan > r0.token_count as usize, ++ &format!( ++ "receipt 0 holds {} tokens in {} particles with a full tail; a lane-wide scan for the \ ++ first PAD stops at {pad_scan}, which is {} tokens PAST receipt 0's end and inside \ ++ receipt 1 ({} tokens) — so a receipt carrying only `first_particle` cannot frame \ ++ itself, and every span whose length is a multiple of 12 is this case rather than a \ ++ corner one. Stated honestly, there are then TWO lawful framings and this probe \ ++ carries both fields: `particle_count` alone bounds the run and, because PAD is a \ ++ RESERVED id, a scan inside that bound is already exact — which costs one \ ++ vocabulary slot; `token_count` costs 4 bytes instead and frees the slot for a full \ ++ 256-id alphabet. What is NOT lawful is inferring the end from padding without a \ ++ bound, which is what this gate measures", ++ r0.token_count, ++ r0.particle_count, ++ pad_scan - r0.token_count as usize, ++ r1.token_count ++ ), ++ ); ++ let v0 = frame_lane.view(&r0, &a).expect("same contract"); ++ let v1 = frame_lane.view(&r1, &a).expect("same contract"); ++ let mut joined = v0.decode(); ++ joined.extend_from_slice(&v1.decode()); ++ g.run( ++ "T-FRAME-ADJ adjacent receipts decode independently and concatenate exactly", ++ v0.len() == exact && v1.len() == full.len() - exact && joined == c0.text.as_bytes(), ++ &format!( ++ "receipt 0 -> {} tokens, receipt 1 -> {} tokens, and their decodes concatenate back to \ ++ the full {} canonical bytes with no bleed in either direction", ++ v0.len(), ++ v1.len(), ++ c0.text.len() ++ ), ++ ); ++ g.run( ++ "T-CONTRACT-GATE a view refuses to open under the wrong contract", ++ frame_lane.view(&r0, &other).is_none(), ++ "`TokenLane::view` compares the receipt's contract id against the codebook offered and \ ++ returns None on mismatch — a mis-read cannot be silent", ++ ); ++ ++ // ---- T-QUERY: query analysis is a pass over QUERY bytes, and it is counted ---- ++ let q_before = query_passes(); ++ let s_before = source_passes(); ++ let store = Arc::new(SeamStore { ++ contract: a.clone(), ++ lane: frame_lane.clone(), ++ }); ++ let mut tk = ReceiptTokenizer::new(Arc::clone(&store), TermMode::Surface); ++ let mut n_q = 0usize; ++ { ++ let mut ts = tk.token_stream("the garden"); ++ while ts.advance() { ++ n_q += 1; ++ } ++ } ++ g.run( ++ "T-QUERY query analysis is separately counted and touches no source", ++ query_passes() - q_before == 1 && source_passes() - s_before == 0 && n_q > 0, ++ &format!( ++ "analysing the query \"the garden\" cost 1 QUERY tokenization ({n_q} terms) and 0 \ ++ source tokenizations. A query is different bytes than the corpus; folding it into \ ++ one number would make the zero-retokenization claim a lie, so the counters are \ ++ separate and both are reported" ++ ), ++ ); ++ ++ // ---- T-FENCE ---- ++ let srcs = [ ++ include_str!("../src/contract.rs"), ++ include_str!("../src/lane.rs"), ++ include_str!("../src/lexical.rs"), ++ include_str!("../src/seam_tantivy.rs"), ++ include_str!("../src/forward.rs"), ++ ]; ++ let code: String = srcs ++ .iter() ++ .map(|s| code_lines(s)) ++ .collect::>() ++ .join("\n"); ++ let manifest = include_str!("../Cargo.toml").to_lowercase(); ++ let no_class_addr = !code.contains("classid") && !code.contains("class_id"); ++ let no_df = !manifest.contains("polars") && !manifest.contains("dataframe"); ++ g.run( ++ "T-FENCE no class address in the token path, no DataFrame in the manifest", ++ no_class_addr && no_df, ++ &format!( ++ "{} lines of non-comment library code contain no `classid`/`class_id` — the contract \ ++ id is a FIELD on the receipt, never smuggled into an address (E-CONTENT-NEVER-\ ++ TRAVELS-IN-CLASSID-1); and the manifest declares no DataFrame of any kind. The \ ++ online path here is parse -> normalise -> tokenize -> index -> project, and not one \ ++ of those five steps is a groupby, a join, a window function or a columnar expression", ++ code.lines().count() ++ ), ++ ); ++ ++ // ---- the measured report ---- ++ println!("\n── corpus / lane ──"); ++ println!( ++ "{:<22} {:>8} {:>7} {:>8} {:>7} {:>6} {:>9} {:>6} {:>5} {:>5} {:>7}", ++ "corpus", ++ "bytes", ++ "spans", ++ "tokens", ++ "uniq", ++ "part.", ++ "resident", ++ "p50", ++ "p95", ++ "max", ++ "cont%" ++ ); ++ // `encode_probes` is #1012's honest cost unit: merge-table probes, never ++ // wall time. A debug/release timing here would be a junk number. ++ for s in &summaries { ++ println!( ++ "{:<22} {:>8} {:>7} {:>8} {:>7} {:>6} {:>9} {:>6} {:>5} {:>5} {:>6.1}%", ++ s.name, ++ s.bytes, ++ s.spans, ++ s.tokens, ++ s.uniq_tokens, ++ s.particles, ++ s.resident_bytes, ++ s.p50, ++ s.p95, ++ s.pmax, ++ s.continuation_rate ++ ); ++ } ++ println!( ++ "encode cost (merge-table probes, not wall time): {}", ++ summaries ++ .iter() ++ .map(|s| format!("{}={}", s.name, s.encode_probes)) ++ .collect::>() ++ .join(" ") ++ ); ++ println!("\n── lexical projection (DeepNSM-v2) ──"); ++ println!( ++ "{:<22} {:>7} {:>9} {:>8} {:>16} {:>10} {:>6}", ++ "corpus", "units", "resolved", "OOV%", "tok/unit p50/95/max", "straddle", "SPO" ++ ); ++ for s in &summaries { ++ println!( ++ "{:<22} {:>7} {:>9} {:>7.1}% {:>10}/{}/{} {:>10} {:>6}", ++ s.name, ++ s.lex_units, ++ s.lex_resolved, ++ 100.0 - pct(s.lex_resolved, s.lex_units), ++ s.tokens_per_unit.0, ++ s.tokens_per_unit.1, ++ s.tokens_per_unit.2, ++ s.multi_unit_tokens, ++ s.spo ++ ); ++ } ++ println!("\n── Tantivy index (same segmentation, two term readings) ──"); ++ println!( ++ "{:<34} {:>5} {:>7} {:>6} {:>12} {:>11} {:>7}", ++ "index", "segs", "terms", "docs", "index tokens", "bytes", "phrase" ++ ); ++ for (name, st) in &index_rows { ++ println!( ++ "{:<34} {:>5} {:>7} {:>6} {:>12} {:>11} {:>7}", ++ name, st.segments, st.terms, st.docs, st.total_tokens, st.bytes, st.phrase_hits ++ ); ++ } ++ println!("\n── forward-prediction input surface (counting BASELINE, not a model) ──"); ++ println!( ++ "{:<22} {:>6} {:>10} {:>11} {:>14}", ++ "corpus", "best k", "top-1", "positions", "DeepNSM cov%" ++ ); ++ for (name, k, acc, pos, cov) in &forward_rows { ++ println!( ++ "{name:<22} {k:>6} {:>9.1}% {pos:>11} {cov:>13.1}%", ++ 100.0 * acc ++ ); ++ } ++ println!("\n── passes ──"); ++ println!( ++ "source tokenizations: {} (= one per span, summed over both corpora and the framing \ ++ fixture)\nquery tokenizations: {} (separate by construction)", ++ source_passes(), ++ query_passes() ++ ); ++ ++ println!("\nPROBE-TOKEN-SEAM-1: ALL {} GATES GREEN", g.pass); ++ println!( ++ "\nverdict: ONE receipt drove all three consumers. Tantivy indexed a receipt HANDLE and \ ++ never received the source; DeepNSM-v2 projected from ids alone through an unmodified \ ++ library; the forward surface is a borrowed slice of the same particles. Byte offsets are \ ++ DERIVED from the codebook's length table, so the receipt stores none. What is NOT settled \ ++ here is stated in the report: the resident carrier is still a probe-local Vec, the OCR \ ++ boundary supplies no byte offsets to attach a span to, the 8-bit vocabulary lane \ ++ saturated at 75 KB, and there is no callable PoS surface anywhere — the module that \ ++ held one was deliberately deleted, and the grounding cited for that deletion is \ ++ itself an example binary outside this repo's dependency barrier." ++ ); ++} +diff --git a/crates/paperless-token/src/contract.rs b/crates/paperless-token/src/contract.rs +new file mode 100644 +index 0000000..e631b26 +--- /dev/null ++++ b/crates/paperless-token/src/contract.rs +@@ -0,0 +1,332 @@ ++//! The immutable, versioned tokenizer contract — a codebook plus its identity. ++//! ++//! ```text ++//! A PERSISTED TOKEN ID IS MEANINGLESS WITHOUT THE CODEBOOK THAT ASSIGNED IT. ++//! CONTENT NEVER TRAVELS IN CLASS ADDRESSES. THE CONTRACT ID IS A FIELD. ++//! ``` ++//! ++//! The BPE trainer/encoder/decoder below is carried unchanged in behaviour from ++//! `lance-graph` `PROBE-TOKEN-BPE-GEOMETRY-1` (#1012, epiphany ++//! `E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1`): base alphabet = the corpus's own ++//! distinct bytes, greedy most-frequent-adjacent-pair merges to a 255 cap, ++//! deterministic tie-break, `0xFF` reserved as PAD. What is NEW here is ++//! everything #1012 deliberately left out of scope for a production carrier: ++//! ++//! - a **contract id** (a digest over the canonical serialisation of the table ++//! plus the normalisation rule id), so a stored id is interpretable; ++//! - a per-id **decoded length** table, which is what makes byte offsets a ++//! derived quantity instead of a stored column; ++//! - a per-id **surface** table, which is what lets every downstream projection ++//! run from token ids alone, never re-reading the source. ++//! ++//! Counters: [`source_passes`] counts tokenizations of SOURCE bytes; ++//! [`query_passes`] counts tokenizations of QUERY bytes. They are separate on ++//! purpose — a query is different bytes, and hiding it inside one number would ++//! make the zero-retokenization gate a lie. ++ ++use std::collections::HashMap; ++use std::sync::atomic::{AtomicUsize, Ordering}; ++ ++use sha2::{Digest, Sha256}; ++ ++/// Reserved id: padding inside a particle. Never emitted by encoding. ++pub const PAD: u8 = 0xFF; ++/// Ids `0..=254` are assignable; `255` is [`PAD`]. ++pub const VOCAB_CAP: usize = 255; ++ ++static SOURCE_PASSES: AtomicUsize = AtomicUsize::new(0); ++static QUERY_PASSES: AtomicUsize = AtomicUsize::new(0); ++ ++/// How many times SOURCE bytes have been tokenized in this process. ++#[must_use] ++pub fn source_passes() -> usize { ++ SOURCE_PASSES.load(Ordering::Relaxed) ++} ++ ++/// How many times QUERY bytes have been tokenized in this process. ++#[must_use] ++pub fn query_passes() -> usize { ++ QUERY_PASSES.load(Ordering::Relaxed) ++} ++ ++/// What an id expands to. ++#[derive(Clone, Copy, Debug)] ++pub enum Expansion { ++ /// A base alphabet byte. ++ Base(u8), ++ /// A merge of two earlier ids. ++ Pair(u8, u8), ++} ++ ++/// The normalisation applied to source bytes BEFORE tokenization. Part of the ++/// contract identity: the same bytes under a different rule are a different ++/// token stream, so the rule id is hashed into the contract id. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub enum NormRule { ++ /// Bytes are taken as-is. Reconstruction is byte-exact. ++ Identity, ++ /// ASCII-lowercased. Reconstruction is exact against the LOWERCASED text, ++ /// which then becomes the canonical text — the authority order is stated ++ /// rather than implied. ++ AsciiLowercase, ++} ++ ++impl NormRule { ++ const fn id(self) -> u8 { ++ match self { ++ Self::Identity => 0, ++ Self::AsciiLowercase => 1, ++ } ++ } ++ ++ /// Apply the rule, producing the canonical bytes. ++ #[must_use] ++ pub fn apply(self, src: &[u8]) -> Vec { ++ match self { ++ Self::Identity => src.to_vec(), ++ Self::AsciiLowercase => src.to_ascii_lowercase(), ++ } ++ } ++} ++ ++/// A trained, immutable tokenizer codebook plus the identity that makes its ++/// output interpretable. ++#[derive(Clone, Debug)] ++pub struct TokenizerContract { ++ expand: Vec, ++ base_of: HashMap, ++ merges: Vec<((u8, u8), u8)>, ++ /// Decoded surface bytes per id. `<= 255` short byte strings. ++ strings: Vec>, ++ /// `strings[id].len()`, hoisted — the table that makes offsets derived. ++ byte_len: Vec, ++ norm: NormRule, ++ contract_id: [u8; 32], ++} ++ ++impl TokenizerContract { ++ /// Train on a byte corpus under `norm`. ++ /// ++ /// # Panics ++ /// Never on a non-empty corpus; an empty corpus yields an empty alphabet ++ /// and `encode` will then reject every byte (see [`Self::try_encode`]). ++ #[must_use] ++ pub fn train(corpus: &[u8], norm: NormRule) -> Self { ++ let corpus = norm.apply(corpus); ++ let corpus = corpus.as_slice(); ++ let mut base_of: HashMap = HashMap::new(); ++ let mut expand: Vec = Vec::new(); ++ let mut strings: Vec> = Vec::new(); ++ for &b in corpus { ++ base_of.entry(b).or_insert_with(|| { ++ expand.push(Expansion::Base(b)); ++ strings.push(vec![b]); ++ u8::try_from(expand.len() - 1).expect("alphabet <= 256 distinct bytes") ++ }); ++ } ++ let mut stream: Vec = corpus.iter().map(|b| base_of[b]).collect(); ++ let mut merges = Vec::new(); ++ while expand.len() < VOCAB_CAP { ++ let mut pf: HashMap<(u8, u8), usize> = HashMap::new(); ++ for w in stream.windows(2) { ++ *pf.entry((w[0], w[1])).or_default() += 1; ++ } ++ // Deterministic tie-break (count desc, then pair asc): the table is ++ // reproducible from the corpus alone, with no external state. ++ let Some((&pair, &count)) = pf ++ .iter() ++ .max_by_key(|(&(a, b), &c)| (c, std::cmp::Reverse((a, b)))) ++ else { ++ break; ++ }; ++ if count < 2 { ++ break; ++ } ++ let id = u8::try_from(expand.len()).expect("bounded by VOCAB_CAP"); ++ expand.push(Expansion::Pair(pair.0, pair.1)); ++ let mut s = strings[pair.0 as usize].clone(); ++ s.extend_from_slice(&strings[pair.1 as usize]); ++ strings.push(s); ++ merges.push((pair, id)); ++ let mut out = Vec::with_capacity(stream.len()); ++ let mut i = 0; ++ while i < stream.len() { ++ if i + 1 < stream.len() && (stream[i], stream[i + 1]) == pair { ++ out.push(id); ++ i += 2; ++ } else { ++ out.push(stream[i]); ++ i += 1; ++ } ++ } ++ stream = out; ++ } ++ let byte_len = strings ++ .iter() ++ .map(|s| u32::try_from(s.len()).expect("token surface is short")) ++ .collect(); ++ let mut me = Self { ++ expand, ++ base_of, ++ merges, ++ strings, ++ byte_len, ++ norm, ++ contract_id: [0; 32], ++ }; ++ me.contract_id = me.compute_contract_id(); ++ me ++ } ++ ++ /// The canonical serialisation the contract id digests. Stated explicitly so ++ /// the identity is reproducible from the table, not from allocation order. ++ fn canonical_bytes(&self) -> Vec { ++ let mut out = Vec::with_capacity(8 + self.expand.len() * 4); ++ out.extend_from_slice(b"PLTOKC01"); ++ out.push(self.norm.id()); ++ out.push(u8::try_from(VOCAB_CAP).expect("255")); ++ out.extend_from_slice( ++ &u32::try_from(self.expand.len()) ++ .unwrap_or(u32::MAX) ++ .to_le_bytes(), ++ ); ++ for e in &self.expand { ++ match *e { ++ Expansion::Base(b) => { ++ out.push(0); ++ out.push(b); ++ out.push(0); ++ } ++ Expansion::Pair(l, r) => { ++ out.push(1); ++ out.push(l); ++ out.push(r); ++ } ++ } ++ } ++ out ++ } ++ ++ fn compute_contract_id(&self) -> [u8; 32] { ++ let mut h = Sha256::new(); ++ h.update(self.canonical_bytes()); ++ h.finalize().into() ++ } ++ ++ /// The contract identity. Two contracts with the same id assign the same ++ /// meaning to the same id; two with different ids do not. ++ #[must_use] ++ pub const fn contract_id(&self) -> [u8; 32] { ++ self.contract_id ++ } ++ ++ /// The contract id, hex, for reports and receipts. ++ #[must_use] ++ pub fn contract_hex(&self) -> String { ++ use std::fmt::Write as _; ++ self.contract_id.iter().fold(String::new(), |mut out, b| { ++ let _ = write!(out, "{b:02x}"); ++ out ++ }) ++ } ++ ++ /// The normalisation rule this contract was trained under. ++ #[must_use] ++ pub const fn norm(&self) -> NormRule { ++ self.norm ++ } ++ ++ /// Number of assigned ids. ++ #[must_use] ++ pub fn vocab_len(&self) -> usize { ++ self.expand.len() ++ } ++ ++ /// Number of merges. ++ #[must_use] ++ pub fn merge_len(&self) -> usize { ++ self.merges.len() ++ } ++ ++ /// The decoded surface of one id. ++ #[must_use] ++ pub fn surface(&self, id: u8) -> &[u8] { ++ &self.strings[id as usize] ++ } ++ ++ /// The decoded byte length of one id. This is the whole reason the receipt ++ /// carries no offset column: an offset is a prefix sum over this table. ++ #[must_use] ++ pub fn byte_len(&self, id: u8) -> u32 { ++ self.byte_len[id as usize] ++ } ++ ++ /// Encode SOURCE bytes. Counted by [`source_passes`]. ++ /// ++ /// Returns `None` if the input contains a byte outside the trained ++ /// alphabet — a contract is only valid for the alphabet it was trained on, ++ /// and silently dropping an unknown byte would break reconstruction. ++ pub fn try_encode(&self, src: &[u8]) -> Option<(Vec, usize)> { ++ SOURCE_PASSES.fetch_add(1, Ordering::Relaxed); ++ self.encode_inner(src) ++ } ++ ++ /// Encode QUERY bytes. Counted by [`query_passes`], never by ++ /// [`source_passes`]. ++ pub fn try_encode_query(&self, q: &[u8]) -> Option<(Vec, usize)> { ++ QUERY_PASSES.fetch_add(1, Ordering::Relaxed); ++ self.encode_inner(q) ++ } ++ ++ fn encode_inner(&self, src: &[u8]) -> Option<(Vec, usize)> { ++ let norm = self.norm.apply(src); ++ let mut stream: Vec = Vec::with_capacity(norm.len()); ++ for b in &norm { ++ stream.push(*self.base_of.get(b)?); ++ } ++ let mut probes = 0usize; ++ for &(pair, id) in &self.merges { ++ let mut out = Vec::with_capacity(stream.len()); ++ let mut i = 0; ++ while i < stream.len() { ++ probes += 1; ++ if i + 1 < stream.len() && (stream[i], stream[i + 1]) == pair { ++ out.push(id); ++ i += 2; ++ } else { ++ out.push(stream[i]); ++ i += 1; ++ } ++ } ++ stream = out; ++ } ++ Some((stream, probes)) ++ } ++ ++ /// Decode ids to canonical bytes. `PAD` is skipped, so a decode is only as ++ /// correct as the framing that told it where to stop — see ++ /// [`crate::lane`]. ++ #[must_use] ++ pub fn decode(&self, tokens: &[u8]) -> (Vec, usize) { ++ let mut out = Vec::new(); ++ let mut steps = 0usize; ++ let mut stack: Vec = Vec::new(); ++ for &t in tokens { ++ if t == PAD { ++ continue; ++ } ++ stack.push(t); ++ while let Some(id) = stack.pop() { ++ steps += 1; ++ match self.expand[id as usize] { ++ Expansion::Base(b) => out.push(b), ++ Expansion::Pair(l, r) => { ++ stack.push(r); ++ stack.push(l); ++ } ++ } ++ } ++ } ++ (out, steps) ++ } ++} +diff --git a/crates/paperless-token/src/forward.rs b/crates/paperless-token/src/forward.rs +new file mode 100644 +index 0000000..1157fce +--- /dev/null ++++ b/crates/paperless-token/src/forward.rs +@@ -0,0 +1,130 @@ ++//! The forward-prediction input surface. ++//! ++//! ```text ++//! TOKEN SEQUENCE = RESIDENT. MODEL STATE = EPHEMERAL. NEITHER OWNS THE OTHER. ++//! ``` ++//! ++//! This module does NOT train a language model, and the counting predictor ++//! below is not one. Its whole job is to answer a narrow question: does the ++//! resident lane hand a forward predictor a usable input surface — ordered ids, ++//! borrowed, with no second population and no separate tokenization? ++//! ++//! [`windows`] is the surface: `(&[u8] context, u8 next)` pairs cut straight out ++//! of the borrowed view. An LSTM would consume exactly this and keep its ++//! weights, hidden state and logits entirely to itself. ++//! ++//! The predictor is an order-k count table so that the surface can be exercised ++//! end to end and reported with a number instead of an assertion. A count table ++//! is a BASELINE, not a claim about what a trained model would achieve; the ++//! probe says so where it prints the number. ++ ++use std::collections::HashMap; ++ ++use crate::lane::TokenStreamView; ++ ++/// Borrowed `(context, next)` windows over a view. Allocation-free. ++pub fn windows<'a>( ++ view: &TokenStreamView<'a>, ++ k: usize, ++) -> impl Iterator + 'a { ++ let ids = view.ids(); ++ (k..ids.len()).map(move |i| (&ids[i - k..i], ids[i])) ++} ++ ++/// An order-k next-token count table. Ephemeral by construction: it is built ++/// from borrowed windows and dropped; nothing about it is resident. ++#[derive(Debug, Default)] ++pub struct CountPredictor { ++ k: usize, ++ table: HashMap, [u32; 256]>, ++ total: HashMap, u32>, ++} ++ ++impl CountPredictor { ++ /// Empty order-k predictor. ++ #[must_use] ++ pub fn new(k: usize) -> Self { ++ Self { ++ k, ++ table: HashMap::new(), ++ total: HashMap::new(), ++ } ++ } ++ ++ /// Observe every window of one view. ++ pub fn observe(&mut self, view: &TokenStreamView<'_>) { ++ for (ctx, next) in windows(view, self.k) { ++ let e = self.table.entry(ctx.to_vec()).or_insert([0; 256]); ++ e[next as usize] += 1; ++ *self.total.entry(ctx.to_vec()).or_insert(0) += 1; ++ } ++ } ++ ++ /// Most likely next id for a context, if the context was seen. ++ /// ++ /// # Panics ++ /// Never: the index comes from `enumerate` over a 256-element array, so the ++ /// `u8` conversion cannot fail. ++ #[must_use] ++ pub fn predict(&self, ctx: &[u8]) -> Option { ++ let counts = self.table.get(ctx)?; ++ let (best, n) = counts ++ .iter() ++ .enumerate() ++ .max_by_key(|(id, &c)| (c, std::cmp::Reverse(*id)))?; ++ (*n > 0).then(|| u8::try_from(best).expect("index < 256")) ++ } ++ ++ /// Distinct contexts observed. ++ #[must_use] ++ pub fn contexts(&self) -> usize { ++ self.table.len() ++ } ++ ++ /// Order. ++ #[must_use] ++ pub const fn order(&self) -> usize { ++ self.k ++ } ++} ++ ++/// The outcome of scoring one held-out set. ++#[derive(Clone, Copy, Debug, Default)] ++pub struct ForwardScore { ++ /// Positions scored. ++ pub scored: usize, ++ /// Positions whose context was never seen in training. ++ pub unseen: usize, ++ /// Correct top-1 predictions. ++ pub hits: usize, ++} ++ ++impl ForwardScore { ++ /// Top-1 accuracy over scored positions. ++ #[must_use] ++ pub fn accuracy(&self) -> f64 { ++ if self.scored == 0 { ++ 0.0 ++ } else { ++ f64::from(u32::try_from(self.hits).unwrap_or(u32::MAX)) ++ / f64::from(u32::try_from(self.scored).unwrap_or(u32::MAX)) ++ } ++ } ++} ++ ++/// Score a predictor over held-out views. ++#[must_use] ++pub fn score(pred: &CountPredictor, views: &[TokenStreamView<'_>]) -> ForwardScore { ++ let mut s = ForwardScore::default(); ++ for v in views { ++ for (ctx, next) in windows(v, pred.order()) { ++ s.scored += 1; ++ match pred.predict(ctx) { ++ Some(p) if p == next => s.hits += 1, ++ Some(_) => {} ++ None => s.unseen += 1, ++ } ++ } ++ } ++ s ++} +diff --git a/crates/paperless-token/src/lane.rs b/crates/paperless-token/src/lane.rs +new file mode 100644 +index 0000000..f49ef41 +--- /dev/null ++++ b/crates/paperless-token/src/lane.rs +@@ -0,0 +1,260 @@ ++//! The resident token lane and its framing. ++//! ++//! ```text ++//! THE POPULATION DOES NOT MOVE. THE VIEW DOES. ++//! TOKEN_COUNT IS AUTHORITATIVE. PAD IS NOT A LENGTH. ++//! ``` ++//! ++//! The lane is a flat `Vec<[u8; 12]>` — the V3 content-blind 12-byte payload ++//! (`lance_graph_contract::facet::FacetCascade` carries exactly this shape: ++//! `facet_classid(4) + 6 x FacetTier{lo,hi} = 16 B`, size-asserted). Twelve ++//! `u8` token ids per particle, two per `(8:8)` tier, never widened to `u16`. ++//! ++//! # Framing: what #1012 left open ++//! ++//! `PROBE-TOKEN-BPE-GEOMETRY-1` measured that EVERY verse of its corpus needed ++//! more than one particle (p50 = 4, max = 8), and refused to pick a framing ++//! mechanism. There is no shipped token continuation field anywhere in ++//! `lance-graph`; the nearest precedent in shape is ++//! `rail_geometry::RailCarving::AxisSlab { reg, cont: Option }`, which ++//! chains ONE register to ONE possibly-discontiguous continuation register and ++//! therefore caps at `RAIL_MAX_DEPTH = 24` levels. That cap is too short here ++//! by construction — a 12-token cap doubled is still under the measured p50 of ++//! 4 particles — so this crate takes the other lawful shape: a **contiguous ++//! run** described by `first_particle + particle_count`, with `token_count` as ++//! the authority. ++//! ++//! `PAD` fills only the tail of the LAST particle of a run and is never ++//! consulted to find a length: a run whose token count is an exact multiple of ++//! 12 contains no PAD at all, and inferring its end from padding would read ++//! straight into the next receipt. The probe exercises exactly that case. ++ ++use crate::contract::{TokenizerContract, PAD}; ++ ++/// Ids per particle: the 12-byte payload, one `u8` per byte. ++pub const IDS_PER_PARTICLE: usize = 12; ++ ++/// The same constant where a receipt's 32-bit fields need it. Spelled out ++/// rather than cast, so no `as` conversion appears on a framing path. ++pub const IDS_PER_PARTICLE_U32: u32 = 12; ++ ++/// The resident particle: the V3 content-blind 12-byte payload. ++pub type TokenParticle = [u8; IDS_PER_PARTICLE]; ++ ++/// A source document's identity. Opaque to this crate — supplied by intake. ++pub type SourceId = u32; ++ ++/// A span within a source (a page, a paragraph, a verse, a cell). ++pub type SpanId = u32; ++ ++/// What one tokenization produced, and everything needed to read it back. ++/// ++/// This is the RECEIPT. It carries no bytes and no offsets: the ids live in the ++/// lane, and offsets are a prefix sum over the contract's per-id length table. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub struct TokenStreamReceipt { ++ /// Which document. ++ pub source_id: SourceId, ++ /// Which span of it. ++ pub span_id: SpanId, ++ /// Which codebook assigned these ids. Without it they are meaningless. ++ pub tokenizer_contract_id: [u8; 32], ++ /// Byte offset of the span within the source's canonical text. ++ pub byte_from: u32, ++ /// AUTHORITATIVE token count. Not derivable from padding. ++ pub token_count: u32, ++ /// Index of the first particle of the run. ++ pub first_particle: u32, ++ /// Number of particles in the run: `ceil(token_count / 12)`. ++ pub particle_count: u32, ++} ++ ++impl TokenStreamReceipt { ++ /// Whether the run's tail is exactly full — the case where PAD-inference ++ /// would silently read into the next receipt. ++ #[must_use] ++ pub const fn tail_is_full(&self) -> bool { ++ self.token_count.is_multiple_of(IDS_PER_PARTICLE_U32) && self.token_count != 0 ++ } ++} ++ ++/// The resident population: particles, plus the receipts that frame them. ++/// ++/// Nothing here owns text. The canonical source text stays authoritative and ++/// lives outside; this lane holds ids and framing only. ++#[derive(Clone, Debug, Default)] ++pub struct TokenLane { ++ particles: Vec, ++ receipts: Vec, ++} ++ ++impl TokenLane { ++ /// Empty lane. ++ #[must_use] ++ pub fn new() -> Self { ++ Self::default() ++ } ++ ++ /// Append one tokenized span. The ids are packed 12 per particle with a PAD ++ /// tail; `token_count` is recorded because the tail is not a length. ++ /// ++ /// # Panics ++ /// If the lane or the span exceeds `u32::MAX` — a receipt addresses the ++ /// lane with 32-bit fields by design, and a silent wrap there would be a ++ /// mis-framed span rather than a large one. ++ pub fn append( ++ &mut self, ++ source_id: SourceId, ++ span_id: SpanId, ++ byte_from: u32, ++ contract: &TokenizerContract, ++ tokens: &[u8], ++ ) -> TokenStreamReceipt { ++ let first_particle = u32::try_from(self.particles.len()).expect("lane fits u32"); ++ for chunk in tokens.chunks(IDS_PER_PARTICLE) { ++ let mut p = [PAD; IDS_PER_PARTICLE]; ++ p[..chunk.len()].copy_from_slice(chunk); ++ self.particles.push(p); ++ } ++ let token_count = u32::try_from(tokens.len()).expect("span fits u32"); ++ let receipt = TokenStreamReceipt { ++ source_id, ++ span_id, ++ tokenizer_contract_id: contract.contract_id(), ++ byte_from, ++ token_count, ++ first_particle, ++ particle_count: u32::try_from(self.particles.len()).expect("lane fits u32") ++ - first_particle, ++ }; ++ self.receipts.push(receipt); ++ receipt ++ } ++ ++ /// Every receipt, in append order. ++ #[must_use] ++ pub fn receipts(&self) -> &[TokenStreamReceipt] { ++ &self.receipts ++ } ++ ++ /// The raw resident particles. Exposed so a probe can demonstrate what a ++ /// PAD-scan would read — the framing falsifier needs to see past the ++ /// receipt's own boundary to prove that `token_count` is load-bearing. ++ #[must_use] ++ pub fn particles(&self) -> &[TokenParticle] { ++ &self.particles ++ } ++ ++ /// Resident particle count. ++ #[must_use] ++ pub fn particle_len(&self) -> usize { ++ self.particles.len() ++ } ++ ++ /// Bytes owned by the resident lane (particles + receipts). ++ #[must_use] ++ pub fn resident_bytes(&self) -> usize { ++ self.particles.len() * IDS_PER_PARTICLE ++ + self.receipts.len() * core::mem::size_of::() ++ } ++ ++ /// A BORROWED view of one receipt's ids. No copy, no allocation: this is a ++ /// slice of the resident population, trimmed by the authoritative ++ /// `token_count` rather than by looking for PAD. ++ /// ++ /// Returns `None` if the contract does not match the receipt — an id is ++ /// only interpretable under the codebook that assigned it. ++ #[must_use] ++ pub fn view<'a>( ++ &'a self, ++ r: &TokenStreamReceipt, ++ contract: &'a TokenizerContract, ++ ) -> Option> { ++ if r.tokenizer_contract_id != contract.contract_id() { ++ return None; ++ } ++ let start = r.first_particle as usize; ++ let end = start + r.particle_count as usize; ++ let flat = self.particles.get(start..end)?; ++ // SAFETY-free reinterpretation: [[u8;12]] is contiguous, so a flat id ++ // slice is a borrow, not a copy. `as_flattened` keeps it in safe Rust. ++ let ids = &flat.as_flattened()[..r.token_count as usize]; ++ Some(TokenStreamView { ++ ids, ++ contract, ++ byte_from: r.byte_from, ++ }) ++ } ++} ++ ++/// One token as the view yields it. Offsets are DERIVED during the walk from ++/// the contract's per-id length table; nothing stored them. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub struct TokenRef { ++ /// Position in the span, in tokens. ++ pub position: u32, ++ /// The id. ++ pub id: u8, ++ /// Byte offset of the token in the source's canonical text. ++ pub byte_from: u32, ++ /// End byte offset (exclusive). ++ pub byte_to: u32, ++} ++ ++/// A borrowed window onto one receipt's ids. Holds no owned token data. ++#[derive(Clone, Copy, Debug)] ++pub struct TokenStreamView<'a> { ++ ids: &'a [u8], ++ contract: &'a TokenizerContract, ++ byte_from: u32, ++} ++ ++impl<'a> TokenStreamView<'a> { ++ /// The borrowed id slice — the input surface a forward predictor consumes. ++ #[must_use] ++ pub const fn ids(&self) -> &'a [u8] { ++ self.ids ++ } ++ ++ /// The contract these ids are read under. ++ #[must_use] ++ pub const fn contract(&self) -> &'a TokenizerContract { ++ self.contract ++ } ++ ++ /// Token count. ++ #[must_use] ++ pub const fn len(&self) -> usize { ++ self.ids.len() ++ } ++ ++ /// Whether the span is empty. ++ #[must_use] ++ pub const fn is_empty(&self) -> bool { ++ self.ids.is_empty() ++ } ++ ++ /// Walk the tokens, deriving offsets by prefix sum. Allocation-free. ++ /// ++ /// # Panics ++ /// If a span exceeds `u32::MAX` tokens; see [`TokenLane::append`]. ++ pub fn tokens(&self) -> impl Iterator + '_ { ++ let mut cursor = self.byte_from; ++ self.ids.iter().enumerate().map(move |(i, &id)| { ++ let from = cursor; ++ cursor += self.contract.byte_len(id); ++ TokenRef { ++ position: u32::try_from(i).expect("span fits u32"), ++ id, ++ byte_from: from, ++ byte_to: cursor, ++ } ++ }) ++ } ++ ++ /// Reconstruct the span's canonical bytes from the ids alone. ++ #[must_use] ++ pub fn decode(&self) -> Vec { ++ self.contract.decode(self.ids).0 ++ } ++} +diff --git a/crates/paperless-token/src/lexical.rs b/crates/paperless-token/src/lexical.rs +new file mode 100644 +index 0000000..6636362 +--- /dev/null ++++ b/crates/paperless-token/src/lexical.rs +@@ -0,0 +1,128 @@ ++//! The `DeepNSM-v2` projection: token ids -> lexical units -> `WordId`. ++//! ++//! ```text ++//! BPE TOKEN IDS ARE SEQUENTIAL. DEEPNSM WORD COORDINATES ARE SEMANTIC. ++//! A PROJECTION IS NOT A SECOND VOCABULARY. ++//! ``` ++//! ++//! `DeepNSM-v2`'s lexical unit is defined by its shipped consumers ++//! (`examples/bible_wave.rs::normalise`, `examples/genre_shapes.rs`): split the ++//! text on WHITESPACE, then within each whitespace token keep only the ASCII ++//! alphabetic bytes, lowercase them, and drop the result if shorter than two ++//! characters. `PaletteVocab::id()` is an exact-match `HashMap` lookup and ++//! normalises nothing — its own doc says "caller lowercases/normalizes". ++//! ++//! The whole point of this module is that the rule above is applied to the ++//! CONTRACT'S PER-ID SURFACE TABLE, never to the source text. The function ++//! signature is the proof: [`project`] takes a [`TokenStreamView`] and no ++//! source bytes at all, so re-reading the source is not merely avoided, it is ++//! unavailable. ++//! ++//! What this module deliberately does NOT do: assign a `WordId` to a BPE token. ++//! That would be a second vocabulary wearing `DeepNSM`'s coordinate system, and ++//! the two id spaces mean different things. Many BPE tokens map to one word; ++//! one token can also contain several words. Both cardinalities are measured ++//! rather than assumed. ++ ++use crate::lane::TokenStreamView; ++ ++/// Minimum surviving length, matching `DeepNSM`'s `normalise`. ++pub const MIN_WORD_LEN: usize = 2; ++ ++/// One lexical unit recovered from the token stream. ++#[derive(Clone, Debug, PartialEq, Eq)] ++pub struct LexicalUnit { ++ /// The normalised surface — lowercase ASCII alphabetic only. ++ pub surface: String, ++ /// Byte offset of the unit in the source's canonical text. ++ pub byte_from: u32, ++ /// End byte offset (exclusive). ++ pub byte_to: u32, ++ /// Index of the first token this unit spans. ++ pub first_token: u32, ++ /// Number of tokens this unit spans (>= 1). ++ pub token_span: u32, ++} ++ ++/// Project a borrowed token view into `DeepNSM` lexical units. ++/// ++/// Reads the contract's per-id surface table and nothing else. The source text ++/// is not a parameter. ++/// ++/// # Panics ++/// If a token surface is longer than `u32::MAX` bytes, which a `<= 255`-entry ++/// BPE table cannot produce. ++#[must_use] ++pub fn project(view: &TokenStreamView<'_>) -> Vec { ++ let contract = view.contract(); ++ let mut out = Vec::new(); ++ // The unit under construction: normalised bytes, plus the byte span of the ++ // whitespace-delimited token it came from, plus the token run it spans. ++ let mut buf: Vec = Vec::new(); ++ let mut span_from = 0u32; ++ let mut span_to = 0u32; ++ let mut first_token = 0u32; ++ let mut open = false; ++ ++ let flush = |buf: &mut Vec, ++ open: &mut bool, ++ span_from: u32, ++ span_to: u32, ++ first_token: u32, ++ last_token: u32, ++ out: &mut Vec| { ++ if *open { ++ if buf.len() >= MIN_WORD_LEN { ++ out.push(LexicalUnit { ++ surface: String::from_utf8(buf.clone()).unwrap_or_default(), ++ byte_from: span_from, ++ byte_to: span_to, ++ first_token, ++ token_span: last_token - first_token + 1, ++ }); ++ } ++ buf.clear(); ++ *open = false; ++ } ++ }; ++ ++ let mut last_token = 0u32; ++ for t in view.tokens() { ++ let surface = contract.surface(t.id); ++ for (k, &b) in surface.iter().enumerate() { ++ let at = t.byte_from + u32::try_from(k).expect("token surface is short"); ++ if b.is_ascii_whitespace() { ++ flush( ++ &mut buf, ++ &mut open, ++ span_from, ++ span_to, ++ first_token, ++ last_token, ++ &mut out, ++ ); ++ } else { ++ if !open { ++ open = true; ++ span_from = at; ++ first_token = t.position; ++ } ++ last_token = t.position; ++ span_to = at + 1; ++ if b.is_ascii_alphabetic() { ++ buf.push(b.to_ascii_lowercase()); ++ } ++ } ++ } ++ } ++ flush( ++ &mut buf, ++ &mut open, ++ span_from, ++ span_to, ++ first_token, ++ last_token, ++ &mut out, ++ ); ++ out ++} +diff --git a/crates/paperless-token/src/lib.rs b/crates/paperless-token/src/lib.rs +new file mode 100644 +index 0000000..bfb8091 +--- /dev/null ++++ b/crates/paperless-token/src/lib.rs +@@ -0,0 +1,39 @@ ++//! `paperless-token` — ONE TOKENIZATION RECEIPT, MANY BORROWED CONSUMERS. ++//! ++//! The bounded seam this crate exists to test: can a single versioned BPE ++//! tokenization of one source span drive lexical retrieval (Tantivy), ++//! lexical/grammar projection (DeepNSM-v2) and a forward-prediction input ++//! surface at the same time, WITHOUT re-tokenizing, without a `DataFrame`, and ++//! without a second cognitive population? ++//! ++//! ```text ++//! ONE SOURCE SPAN -> ONE TOKENIZATION RECEIPT. ++//! TOKENIZE ONCE. PROJECT MANY TIMES. ++//! THE POPULATION DOES NOT MOVE. THE VIEW DOES. ++//! TANTIVY IS AN INDEX, NOT MEMORY OWNERSHIP. ++//! BPE IS TOKENIZATION, NOT ONTOLOGY. ++//! CONTENT NEVER TRAVELS IN CLASS ADDRESSES. ++//! ``` ++//! ++//! Layering, and what each layer is forbidden from doing: ++//! ++//! | module | owns | never | ++//! |---|---|---| ++//! | [`contract`] | the codebook + its identity | reads the lane | ++//! | [`lane`] | resident particles + framing | owns text | ++//! | [`lexical`] | the `DeepNSM` projection | reads the source | ++//! | [`seam_tantivy`] | the index seam | owns offsets | ++//! | [`forward`] | the prediction input surface | owns the sequence | ++//! ++//! Status: PROBE. Nothing here is a production carrier. #1012 returned ++//! CAN-FIT, NOT YET BUY at fixture scale; this crate answers the integration ++//! half of the question that verdict left open, and the probe reports what it ++//! measured rather than what it hoped. ++ ++#![forbid(unsafe_code)] ++ ++pub mod contract; ++pub mod forward; ++pub mod lane; ++pub mod lexical; ++pub mod seam_tantivy; +diff --git a/crates/paperless-token/src/seam_tantivy.rs b/crates/paperless-token/src/seam_tantivy.rs +new file mode 100644 +index 0000000..40f844e +--- /dev/null ++++ b/crates/paperless-token/src/seam_tantivy.rs +@@ -0,0 +1,204 @@ ++//! The Tantivy seam: a `Tokenizer` that reads the resident lane instead of text. ++//! ++//! ```text ++//! AN INDEX MAY ACCELERATE THE ABI. IT MUST NEVER BECOME THE ABI. ++//! ``` ++//! ++//! # Why this cannot silently re-tokenize ++//! ++//! The value put into the indexed text field is not the document text. It is a ++//! RECEIPT HANDLE (`rcpt:`). The tokenizer resolves the handle against the ++//! lane and walks borrowed ids. Re-tokenizing the source is therefore not ++//! merely avoided by discipline — Tantivy is never handed the source at all. ++//! ++//! `Tokenizer::token_stream<'a>(&'a mut self, text: &'a str)` places no ++//! requirement that emitted tokens derive from `text`; the only real bounds are ++//! `'static + Clone + Send + Sync`, which is why the store is behind an `Arc`. ++//! ++//! # What Tantivy actually keeps ++//! ++//! Measured in this fork: the indexer reads `Token::text` (into the term bytes) ++//! and `Token::position` (persisted as the position list), uses ++//! `position_length` transiently to advance the position cursor, and NEVER ++//! reads `offset_from`/`offset_to` outside its own tests ++//! (`src/postings/postings_writer.rs::index_text`). Byte offsets are consumed ++//! only by snippet generation, which re-tokenizes the STORED text at query time ++//! (`src/snippet/mod.rs:211`). Two consequences, both stated rather than ++//! discovered later: ++//! ++//! - the index structurally cannot become the owner of offsets, which is ++//! exactly the demarcation this architecture wants; ++//! - Tantivy's built-in snippet generator does not work over a receipt handle. ++//! Highlighting has to be served from the canonical text through the receipt ++//! — which is where it belongs, and is a named consequence, not a defect. ++//! ++//! # The alternative that was measured and rejected for the resident path ++//! ++//! `PreTokenizedString { text: String, tokens: Vec }` allocates one ++//! `String` per token, and `segment_writer.rs` deep-clones the whole boxed ++//! value before indexing it (`PreTokenizedStream::from(*tok_str.clone())`), so ++//! one field costs about `4 + 2N` allocations for N tokens. That is the ++//! materialised token-object population the root memory law forbids. The ++//! tokenizer below reuses ONE `Token` buffer, the way Tantivy's own ++//! `SimpleTokenizer` does. ++ ++use std::sync::Arc; ++ ++use tantivy::tokenizer::{Token, TokenStream, Tokenizer}; ++ ++use crate::contract::TokenizerContract; ++use crate::lane::TokenLane; ++ ++/// The prefix that marks an indexed field value as a receipt handle. ++pub const HANDLE_PREFIX: &str = "rcpt:"; ++ ++/// What a Tantivy term IS. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub enum TermMode { ++ /// The term is the token's decoded surface. Ordinary lexical retrieval, ++ /// with sub-word semantics: a term can be a word fragment. ++ Surface, ++ /// The term is the token id itself, rendered as two hex digits. The term ++ /// dictionary is then bounded by the vocabulary (<= 255 terms) regardless ++ /// of corpus size, and phrase queries run over id sequences. ++ TokenId, ++} ++ ++/// The resident side of the seam: one contract, one lane. ++#[derive(Debug)] ++pub struct SeamStore { ++ /// The codebook every id in the lane is read under. ++ pub contract: TokenizerContract, ++ /// The resident particles and their receipts. ++ pub lane: TokenLane, ++} ++ ++/// Format the field value for a receipt index. ++#[must_use] ++pub fn handle(receipt_index: usize) -> String { ++ format!("{HANDLE_PREFIX}{receipt_index}") ++} ++ ++/// A `Tokenizer` that yields the resident lane's ids for a receipt handle, and ++/// falls back to encoding the QUERY when handed anything else. ++#[derive(Clone)] ++pub struct ReceiptTokenizer { ++ store: Arc, ++ mode: TermMode, ++ token: Token, ++} ++ ++impl ReceiptTokenizer { ++ /// Build over a shared store. ++ #[must_use] ++ pub fn new(store: Arc, mode: TermMode) -> Self { ++ Self { ++ store, ++ mode, ++ token: Token::default(), ++ } ++ } ++} ++ ++/// Where a stream's ids come from. ++enum Ids<'a> { ++ /// Borrowed straight out of the resident lane — the indexing path. ++ Resident(&'a [u8]), ++ /// Owned, because this was a query, whose bytes are not in the lane. ++ Query(Vec), ++} ++ ++impl Ids<'_> { ++ fn as_slice(&self) -> &[u8] { ++ match self { ++ Ids::Resident(s) => s, ++ Ids::Query(v) => v.as_slice(), ++ } ++ } ++} ++ ++/// The stream. Holds one mutable `Token` buffer, reused for every token. ++pub struct ReceiptTokenStream<'a> { ++ ids: Ids<'a>, ++ contract: &'a TokenizerContract, ++ mode: TermMode, ++ token: &'a mut Token, ++ next: usize, ++ cursor: u32, ++} ++ ++impl Tokenizer for ReceiptTokenizer { ++ type TokenStream<'a> = ReceiptTokenStream<'a>; ++ ++ fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> { ++ // Disjoint field borrows: the token buffer is borrowed mutably while ++ // the store is borrowed immutably. ++ let Self { store, mode, token } = self; ++ token.reset(); ++ let contract = &store.contract; ++ let ids = text ++ .strip_prefix(HANDLE_PREFIX) ++ .and_then(|n| n.parse::().ok()) ++ .and_then(|i| store.lane.receipts().get(i)) ++ .and_then(|r| store.lane.view(r, contract)) ++ .map_or_else( ++ || { ++ // Not a handle: this is a QUERY. Encoding it is a pass over ++ // QUERY bytes, counted separately from source passes. ++ let owned = contract ++ .try_encode_query(text.as_bytes()) ++ .map(|(t, _)| t) ++ .unwrap_or_default(); ++ Ids::Query(owned) ++ }, ++ |v| Ids::Resident(v.ids()), ++ ); ++ ReceiptTokenStream { ++ ids, ++ contract, ++ mode: *mode, ++ token, ++ next: 0, ++ cursor: 0, ++ } ++ } ++} ++ ++impl TokenStream for ReceiptTokenStream<'_> { ++ fn advance(&mut self) -> bool { ++ let ids = self.ids.as_slice(); ++ let Some(&id) = ids.get(self.next) else { ++ return false; ++ }; ++ let len = self.contract.byte_len(id); ++ self.token.text.clear(); ++ match self.mode { ++ TermMode::Surface => { ++ let s = String::from_utf8_lossy(self.contract.surface(id)); ++ self.token.text.push_str(&s); ++ } ++ TermMode::TokenId => { ++ use std::fmt::Write as _; ++ let _ = write!(self.token.text, "{id:02x}"); ++ } ++ } ++ self.token.position = self.next; ++ self.token.position_length = 1; ++ // Offsets are derived here and handed over even though this fork's ++ // indexer ignores them: the stream HAS them, and saying so is cheaper ++ // than someone later concluding the seam lost them. ++ self.token.offset_from = self.cursor as usize; ++ self.token.offset_to = (self.cursor + len) as usize; ++ self.cursor += len; ++ self.next += 1; ++ true ++ } ++ ++ fn token(&self) -> &Token { ++ self.token ++ } ++ ++ fn token_mut(&mut self) -> &mut Token { ++ self.token ++ } ++} +diff --git a/docs/TOKEN-SEAM-ARCHITECTURE.md b/docs/TOKEN-SEAM-ARCHITECTURE.md +new file mode 100644 +index 0000000..5964c33 +--- /dev/null ++++ b/docs/TOKEN-SEAM-ARCHITECTURE.md +@@ -0,0 +1,500 @@ ++# The token seam — one tokenization receipt, many borrowed consumers ++ ++**Status:** bounded architecture + probe. `PROBE-TOKEN-SEAM-1`, 37 gates, 9 ++disable-runs verified red-then-green. Nothing here is a production carrier. ++ ++**The question, exactly as posed.** Can Tantivy + one versioned BPE ++`TokenStreamView` become the shared lexical intake seam for document retrieval, ++DeepNSM-v2 reasoning and forward autocomplete, while structured evidence enters ++lance-graph directly and Polars disappears from the online path unless ++measurement proves it is genuinely needed? ++ ++**The answer.** Yes for the seam, measured end to end. The three consumers ran ++off ONE tokenization per span and added ZERO further tokenizations — not by ++discipline but by construction: Tantivy is handed a receipt handle instead of ++text, and the DeepNSM projection's function signature has no source parameter. ++Neither Tantivy nor DeepNSM-v2 needed a single line changed. Polars is not in ++the online path and never was — there is nothing to remove, which is a weaker ++and more honest headline than the one the framing invited. ++ ++What is NOT settled is listed as eight named gaps in §7. Two of them are ++load-bearing: the OCR boundary hands over no byte offsets to attach a span to ++(G1), and the 8-bit vocabulary lane saturates at 75 KB of ordinary English ++(G6). Neither is a defect in the seam; both are the next thing to measure. ++ ++--- ++ ++## 1. The shape ++ ++``` ++ canonical text (AUTHORITATIVE) ++ │ ++ │ exactly one pass per span ++ ▼ ++ versioned BPE contract ++ (codebook + normalisation rule + contract id) ++ │ ++ ▼ ++ resident lane: [u8;12] particles + receipts ++ │ ++ ┌────────────────────┼────────────────────┐ ++ │ borrowed │ borrowed │ borrowed ++ ▼ ▼ ▼ ++ Tantivy DeepNSM-v2 forward predictor ++ terms + positions WordId → PoS → SPO (context, next) windows ++ DERIVED, deletable DERIVED, deletable EPHEMERAL, owns nothing ++``` ++ ++and, in parallel and never through this seam: ++ ++``` ++ structured rows → typed intake (arm-discovery: FeatureSpec + Vec>) ++ → SPO / evidence / provenance ++``` ++ ++The two meet at shared `(source_id, span_id, byte_from)` identity — not at a ++shared table. ++ ++| layer | owns | never | ++|---|---|---| ++| `contract` | the codebook and its identity | reads the lane | ++| `lane` | resident particles + framing | owns text | ++| `lexical` | the DeepNSM projection | reads the source | ++| `seam_tantivy` | the index seam | owns offsets | ++| `forward` | the prediction input surface | owns the sequence | ++ ++Each "never" is proven by a gate, not asserted — §8 lists which disable turns ++each one red. ++ ++## 2. What was measured ++ ++Two committed, real, hermetic corpora. `kjv-genesis-scene` is the same in-tree ++text `PROBE-TOKEN-BPE-GEOMETRY-1` (#1012) used, carried verbatim so the two ++probes are comparable; `alice-paragraphs` is Project Gutenberg's *Alice's ++Adventures in Wonderland*, carried from `tantivy/benches/alice.txt`. ++ ++### lane ++ ++| corpus | bytes | spans | tokens | ratio | uniq ids | particles | resident B | particles/span p50/p95/max | continuation | ++|---|---|---|---|---|---|---|---|---|---| ++| kjv-genesis-scene | 1 126 | 8 | 354 | 3.18× | 137 | 32 | 832 | 4 / 8 / 8 | 87.5 % | ++| alice-paragraphs | 75 514 | 300 | 37 149 | 2.03× | 247 | 3 231 | 55 572 | 8 / 30 / 43 | 100 % | ++ ++Encode cost, in #1012's honest unit (merge-table probes, never wall time): ++86 640 and 7 929 358. ++ ++**The resident lane is ~74 % of the source text, not a fraction of it.** For ++Alice the particles alone are 38 772 B (51 % of source) and the receipts add ++16 800 B. At these span sizes the FRAMING, not the payload, is where the bytes ++go: a 56-byte receipt against 12-byte particles is 30 % of the resident total ++at paragraph granularity and 54 % at verse granularity. #1012 could not see ++this — it had no receipt. ++ ++**Continuation is the norm, confirmed at a second scale.** #1012 measured every ++verse overflowing one particle; here every one of 300 paragraphs does, with a ++p95 of 30 particles. Any production design budgets continuation from the start. ++ ++### the three consumers ++ ++| consumer | added source tokenizations | added query tokenizations | evidence | ++|---|---|---|---| ++| Tantivy (both term modes, both corpora) | 0 | 0 | indexed value is `rcpt:`, never text | ++| DeepNSM-v2 projection | 0 | 0 | `project()` takes no source parameter | ++| forward windows | 0 | 0 | slices pointer-identical to the lane | ++ ++Whole-run totals: **313 source tokenizations** — 8 + 300, one per span, plus 5 ++by deliberate fixtures inside the contract and framing gates — and **1 query ++tokenization**, counted on a separate counter because a query is different ++bytes and folding it into one number would make the claim a lie. ++ ++### Tantivy, driven by the receipt ++ ++| index | segments | terms | docs | index tokens | bytes | phrase hits | ++|---|---|---|---|---|---|---| ++| kjv / surface | 1 | 137 | 8 | 354 | 2 913 | 1 | ++| kjv / token-id | 1 | 137 | 8 | 354 | 2 334 | 1 | ++| alice / surface | 1 | 244 | 300 | 37 149 | 82 904 | 2 | ++| alice / token-id | 1 | 247 | 300 | 37 149 | 81 897 | 2 | ++ ++Three things fall out of that table. ++ ++- **The term dictionary is bounded by the vocabulary, not by the corpus.** 247 ++ terms for 75 KB, because a term is a BPE token. That is a different retrieval ++ regime from word indexing and it is a trade, not a win: recall rises (a query ++ fragment matches inside a word) and per-term selectivity falls. ++- **Surface terms silently merge distinct ids.** Alice indexes 247 ids as 244 ++ surface terms: three ids decode to a string another id also decodes to (BPE ++ can reach the same string by different merge paths). Id-mode keeps them ++ apart, and is also smaller on disk — two-hex-digit terms beat surfaces. ++- **Positions are the receipt's positions.** Asserted directly: the analyzer ++ registered on the index yields `0..token_count` in order for the receipt ++ under test. There is one segmentation and the index consumed it. ++ ++**Tantivy structurally cannot become the owner of offsets.** In this fork the ++indexer's `index_text` reads `Token::text` and `Token::position`, uses ++`position_length` transiently, and reads `offset_from`/`offset_to` NOWHERE ++outside its own tests (`src/postings/postings_writer.rs`). Byte offsets are ++consumed only by snippet generation, which re-tokenizes the STORED text at ++query time (`src/snippet/mod.rs:211`). That is the demarcation this ++architecture wants, handed to us by the index's own design — and it has a ++price, recorded as G7: under the handle design Tantivy's built-in highlighter ++has nothing to highlight, so snippets must be served from the canonical text ++through the receipt. ++ ++The route NOT taken is worth recording. `PreTokenizedString { text: String, ++tokens: Vec }` allocates a `String` per token, and `segment_writer.rs` ++deep-clones the whole boxed value before indexing it — about `4 + 2N` ++allocations for N tokens. That is precisely the materialised token-object ++population the root memory law forbids. The custom tokenizer reuses ONE `Token` ++buffer, the way Tantivy's own `SimpleTokenizer` does. ++ ++### DeepNSM-v2, projected from ids alone ++ ++| corpus | lexical units | resolved to a WordId | OOV | tokens/unit p50/p95/max | straddling tokens | SPO triples | ++|---|---|---|---|---|---|---| ++| kjv-genesis-scene | 225 | 186 | 17.3 % | 1 / 4 / 7 | 30 | 4 | ++| alice-paragraphs | 13 108 | 10 586 | 19.2 % | 2 / 6 / 15 | 58 | 671 | ++ ++**The cardinality is not 1:1 in either direction, measured.** A lexical unit ++spans a median of 2 tokens and up to 15; and 58 tokens carry the start of more ++than one unit — a single BPE token straddling a word boundary. So BPE sequence ++identity and the DeepNSM word coordinate are different id spaces, and the ++projection is a real function rather than a relabelling. Nothing in the seam ++assigns a `WordId` to a BPE token; the ids that reach the FSM are DeepNSM's ++own, resolved from the reconstructed surface against its own frequency-ranked ++vocabulary. ++ ++**DeepNSM-v2 needed no change.** Its library is already tokenizer-free: ++`parse_to_spo(&[Tagged])` consumes `(WordId, Pos)` pairs and touches no string. ++The `split_whitespace`/`normalise` logic lives only in its examples. That is ++the single most load-bearing fact in this report — the seam is adoptable ++because the consumer was already shaped for it. ++ ++The OOV figure is against an 18 559-word academic vocabulary (20 845 COCA rows, ++2 286 of them duplicate surface forms). ~19 % is what an academic word list ++does on Victorian prose; it is a property of the vocabulary, not of the seam. ++ ++### the forward surface ++ ++| corpus | best order | top-1 | k=1 / k=2 / k=3 | positions scored | unseen context | positions with a DeepNSM coordinate | ++|---|---|---|---|---|---|---| ++| kjv-genesis-scene | 1 | 0.0 % | 0.0 / 0.0 / 0.0 | 81 | 19 | 95.8 % | ++| alice-paragraphs | 2 | 26.4 % | 13.3 / 26.4 / 25.4 | 5 093 | 399 | 72.0 % | ++ ++This is a **counting baseline, not a language model**, and it is here to prove ++the input surface rather than to predict anything. Two readings are legitimate: ++the windows are slices INTO the resident particle array (asserted ++pointer-identical, and a disable that copies them turns the gate red), and the ++sequence has enough structure for order-2 context to double order-1 accuracy. ++The KJV row is 0.0 % on 81 scored positions — at 6 training verses that is ++indistinguishable from chance and is reported rather than dropped. ++ ++Of the three candidate input representations named in the brief: **(A)** the ++token id is present by construction; **(B)** the DeepNSM `(basin, identity)` ++palette coordinate is available for 72–96 % of token positions, so the hybrid ++**(C)** is CONSTRUCTIBLE. Which one a trained model should prefer is **not ++measured here and no claim is made** — that needs training, a real corpus, and ++its own probe. ++ ++## 3. The ten answers ++ ++**1. Can one versioned BPE tokenization receipt drive Tantivy directly?** ++Yes, and with no patch to Tantivy. A custom `Tokenizer` reads the resident lane ++and yields borrowed ids; the indexed field value is a receipt handle, so the ++index is never handed the source at all. Indexing 308 documents across four ++index builds added 0 tokenizations of either kind. Positions in the index ARE ++the receipt's positions. Costs: sub-word term semantics, and no built-in ++snippets (G7). ++ ++**2. Can the same receipt project into DeepNSM-v2 without raw-source ++re-tokenization?** Yes. `project()` takes a borrowed view and no source bytes — ++re-reading the source is unavailable, not merely avoided. It reads the ++contract's per-id surface table, which is at most 255 short byte strings. ++DeepNSM-v2's library is unchanged. The one thing the seam is forced to ++duplicate is the COCA part-of-speech mapping — and the reason is a deliberate ++earlier deletion, not an oversight (G3). ++ ++**3. Can the same ordered token ids serve as forward-prediction input?** Yes, ++as borrowed slices of the resident particles. Order-2 context reaches 26.4 % ++top-1 against 13.3 % at order 1 on held-out spans with a counting baseline. The ++predictor owns nothing and is dropped. ++ ++**4. What production framing/codebook metadata is missing from #1012's ++fixture-scale `[u8;12]` result?** Five things, all now specified and gated: ++- a **contract id** — a digest over the canonical serialisation of the table ++ AND the normalisation rule id. Without it a stored `u8` is not a weak ++ reference, it is a wrong one: decoding corpus A's ids under corpus B's ++ codebook returned 551 bytes of garbage where 1 126 were expected. ++- a **framing triple** `first_particle + particle_count + token_count`. There ++ is no shipped token continuation mechanism anywhere in lance-graph; the ++ nearest precedent in shape, `rail_geometry::RailCarving::AxisSlab { reg, cont ++ }`, chains one register to one continuation and caps at 24 levels — under ++ the measured p50 of 4 particles, so it does not fit. Honestly stated, TWO ++ framings are lawful: `particle_count` alone bounds the run and a PAD scan ++ inside that bound is exact because PAD is RESERVED (cost: one vocabulary ++ slot); or `token_count` carries the length in 4 bytes and frees the slot for ++ a full 256-id alphabet. What is unlawful is inferring the end from padding ++ with no bound — measured, a lane-wide PAD scan overshoots receipt 0 by 10 ++ tokens straight into receipt 1, and every span whose length is a multiple of ++ 12 is that case rather than a corner one. ++- a **per-id decoded-length table**, which is what makes byte offsets a DERIVED ++ quantity. The receipt stores no offset column at all; an offset is a prefix ++ sum taken during the walk, checked here against a decode-the-prefix ground ++ truth. ++- a **per-id surface table**, which is what lets every downstream projection ++ run without the source. ++- **span identity** `(source_id, span_id, byte_from)` — and the measured ++ consequence that at these span sizes the receipt is 30–54 % of resident ++ bytes, so the receipt's own layout matters more than the particle's. ++ ++**5. Does any online step still require Polars?** No — and the honest form of ++that answer is that it never did. A sweep of nine checkouts found **zero** ++occurrences of `polars` in any manifest or source file; every `DataFrame` ++mention is prose (a "Kuzu/Polars pattern" note, an unimplemented R-bridge ++sketch, and Tantivy's own columnar test naming). `paperless-rs` and ++`tesseract-rs` — the two repos that constitute this online path — declare none ++of `arrow`, `datafusion`, `lance`, `lancedb`. The online steps are: parse → ++normalise → tokenize → index → project. Not one is a groupby, a join, a window ++function, or a columnar expression. Where a DataFrame WOULD genuinely be ++reached for — structured evidence — the typed path already exists and is not ++tabular algebra either: `lance-graph-arm-discovery` takes `Dataset { spec: ++FeatureSpec, rows: Vec> }`, category-index rows against a schema, with ++its own module doc stating "no one-hot float vector and no embedding". The ++remaining honest gap is that "parse a table" has no in-tree answer at all (no ++`calamine`, no `csv` anywhere), and the answer to that is a parser, not a ++DataFrame (G8). ++ ++**6. What remains authoritative?** In order, and the order is enforced rather ++than described: ++`canonical source text` (authoritative) → `contract` (authoritative for what an ++id MEANS) → `receipt + lane` (exact and reconstructible; byte-identical decode ++on both corpora) → `Tantivy index` (derived, deletable) → `DeepNSM projection` ++(derived, deletable) → `model state` (ephemeral, owns nothing). ++ ++**7. Can Tantivy be deleted and rebuilt without semantic loss?** Yes. It holds ++terms and positions derived from the lane, stores a handle rather than text, ++and persists no offsets. Rebuilding is a re-walk of the lane at 0 source ++tokenizations. What is lost on deletion is query latency — and, under this ++design, the built-in highlighter (G7). ++ ++**8. Can the LSTM be deleted and rebuilt without token or document loss?** ++Yes, trivially: it owns nothing. Windows are borrowed, the predictor is built ++from them and dropped. Weights, hidden state and logits never enter the lane. ++ ++**9. Can the token stream be consumed entirely as borrowed/resident SoA views ++after intake?** As BORROWED views, yes — every consumer here takes ++`&TokenStreamView` and the forward windows are pointer-identical to the lane's ++storage. As a lawful **resident SoA lane**, not yet: this lane is a probe-local ++`Vec<[u8;12]>`. A production lane must either implement `SoaEnvelope` ++(`ColumnDescriptor`, `ENVELOPE_LAYOUT_VERSION = 2`, `verify_layout`, ++`mailbox_owner`) or land as a new `ValueTenant` — whose enum currently has 16 ++variants and none for tokens. That is the boundary, stated rather than crossed ++(G2). ++ ++**10. Does the combined path reduce representation changes versus the ++Pandas/Polars + Elasticsearch style pipeline?** Yes, and it is countable rather ++than rhetorical. ++ ++| | representation changes | tokenizations of the source | ++|---|---|---| ++| classic: text → DataFrame → analyzer → embeddings → ANN → graph adapter | 5 | ≥ 2 (ingest + the engine's own analyzer) | ++| this seam: text → ids → {terms, lexical units, windows} | 1 owned + 3 derived views | **1**, asserted per span | ++ ++The measured figure is 313 source tokenizations for 308 spans plus 5 by ++deliberate fixtures, and 0 added by any consumer. ++ ++## 4. Falsifier verdicts ++ ++| # | falsifier | verdict | ++|---|---|---| ++| F1 | Tantivy requires independent tokenization | **REFUTED** — 0 tokenizations across 4 index builds; the value is a handle | ++| F2 | codebook identity not frozen/versioned | **REFUTED** — contract id digests the table AND the rule; `T-CONTRACT-RULE` | ++| F3 | continuation cannot reconstruct boundaries | **REFUTED** — adjacent receipts decode independently and concatenate exactly | ++| F4 | DeepNSM cannot project without re-tokenizing the source | **REFUTED** — the projection has no source parameter | ++| F5 | the DeepNSM vocabulary is weakened to accommodate BPE | **REFUTED** — crate unchanged; the ids that reach the FSM are its own | ++| F6 | the LSTM needs a second canonical population | **REFUTED** — windows pointer-identical to the lane | ++| F7 | embeddings become canonical identity | **NOT REACHED** — no embedding, ANN or vector store exists here; the cam96 codebook is ABSENT | ++| F8 | Tantivy becomes the only owner of positions/offsets | **STRUCTURALLY IMPOSSIBLE** — the indexer never reads offsets at all | ++| F9 | deleting Tantivy loses canonical state | **REFUTED** — the index is derived from the lane | ++| F10 | structured evidence flattened into text | **NOT EXERCISED** — no structured corpus is present; MecCog is absent | ++| F11 | a DataFrame survives in the online path | **REFUTED** — zero occurrences, workspace-wide | ++| F12 | BPE ids or tokenizer content leak into a class address | **REFUTED** — `T-FENCE` over 730 lines of non-comment code | ++| F13 | source reconstruction is not exact | **REFUTED** — byte-exact on both corpora, both framings | ++| F14 | offsets drift between Tantivy, DeepNSM and the receipt | **REFUTED** — positions asserted equal; spans re-normalise to the source; Tantivy holds no offsets to drift | ++| F15 | one source is tokenized more than once | **REFUTED** — 1 per span, asserted per corpus | ++| F16 | the project expands into a generic LLM/embedding architecture | **HELD** — no model, no embedding, no vector database, no token service | ++ ++F7 and F10 are honest non-results, not passes. F16 is a boundary this document ++is responsible for keeping. ++ ++## 5. What the seam costs, stated plainly ++ ++- The resident lane is **~74 % of the source text** at a 255-id vocabulary — ++ not a compression story. Compression fell from 3.18× on 1 KB to 2.03× on ++ 75 KB as the vocabulary saturated. ++- The Tantivy index for Alice is **82 904 B against 75 514 B of source** — ++ 110 %. An index is an index. ++- Framing overhead is **30–54 % of resident bytes** at paragraph and verse ++ granularity respectively. ++- Encode cost is quadratic-ish in the naive form used here: 7.9 M merge-table ++ probes for 75 KB, because encoding applies all 255 merges as full passes. A ++ production encoder uses a priority queue; this one was carried unchanged from ++ #1012 deliberately, so the two probes' numbers stay comparable. ++ ++None of these are arguments against the seam. They are the numbers a production ++decision needs and did not have. ++ ++## 6. What this does NOT change ++ ++- **HHTL is address geometry; BPE is tokenization.** #1012 measured that a BPE ++ merge tree is pair-ENCODABLE but is not a lawful radix prefix partition ++ (same-depth tokens are prefixes of one another). Nothing here revisits that. ++- **Content never travels in a class address.** The contract id is a FIELD on ++ the receipt. `T-FENCE` greps the library's own non-comment source for ++ `classid`/`class_id` and finds none. ++- **Tokenization is not span-local.** Encoding the KJV text whole yields 346 ++ tokens; encoding it as 8 spans yields 354. Merges cross span boundaries, so ++ the span partition is part of what the receipt identifies. Re-spanning a ++ document changes its ids and therefore its index and its projection — a ++ property to pin, not a bug (G5). ++ ++## 7. The named gaps ++ ++**G1 — the OCR boundary supplies no byte offsets, and this is the one that ++blocks real documents.** `doc.v1` carries `bbox`, `conf`, `leading_space` and a ++page-level `plain_text`, and no `offset`/`span`/`start`/`end` field on any of ++`DocWord`/`DocLine`/`DocPage`. The word→text mapping is *derivable* by replaying ++the documented join rule, but nothing stores it — so intake would have to ++re-implement a rule that lives in `tesseract-ocr/src/renderer.rs`, which is a ++second implementation of one rule and will drift. The fix belongs upstream and ++is small: emit the per-word byte offset where the join is performed, since that ++is the one place the rule exists. ++ ++**G2 — no resident SoA carrier.** The lane is a probe-local `Vec`. Lawful ++options: implement `SoaEnvelope`, or mint a new `ValueTenant` (16 variants ++today, none for tokens). The measured framing overhead says the receipt's ++column layout is the thing to design first. ++ ++**G3 — there is no shipped, callable part-of-speech surface anywhere, and the ++reason is a decision already taken.** `coca_pos`, `archaic_pos` and `normalise` ++are byte-identical in BOTH of DeepNSM-v2's examples, and the doc comment above ++them explains why: *"`deepnsm_v2::lexicon` was deleted after an audit found the ++planner's `insight_coca_read` already grounds this in the master COCA ++`lexicon.tsv` (with lemmatisation)."* Re-adding the module would re-litigate ++that audit, so this seam did not — it restates the minimal tagger and says so. ++ ++But the grounding the deletion relied on does not hold for a lean consumer, and ++that is the finding rather than the complaint. `insight_coca_read` is itself an ++**example binary** in `lance-graph-planner` — not a library API — and that ++crate pulls `serde`/`serde_yml`/`tokio`/`ndarray`, so it is outside the BBB ++dependency set a document-intake binary can carry. Its master `lexicon.tsv` is ++absent from this checkout. So a consumer has exactly three options, none of ++which is "call the shipped one": depend on the planner (violates the barrier), ++restate the minimal tagger (what this seam does, ~20 lines, example-grade), or ++re-open the deletion with this consumer as the new evidence. The third is the ++right one if a second consumer ever needs it; one consumer restating twenty ++lines is not yet a case. ++ ++**G4 — the semantic half is unexercised.** `cam96_codebook.bin` / ++`cam96_codes.bin` are release assets, absent here, so DeepNSM's palette256² ++DISTANCE was not run. Only the lexical/grammar half was. ++ ++**G5 — re-spanning changes the tokens** (see §6). Needs a pin. ++ ++**G6 — the 8-bit lane saturates, and this is the biggest unknown.** Alice used ++247 of 255 ids on 75 KB; the vocabulary was full and compression was already ++falling. The canon's answer is the hi byte of each `(8:8)` pair as a PAGE lane — ++two separate bytes, never a widened `u16` — which is untested. That is the next ++probe, and until it runs no scale claim should be made. ++ ++**G7 — snippets.** Under the handle design Tantivy's highlighter has no text to ++highlight; highlighting must be served from the canonical text through the ++receipt. ++ ++**G9 — the projection's whitespace rule is ASCII where DeepNSM's is Unicode.** ++`project()` walks bytes and splits on `u8::is_ascii_whitespace`; DeepNSM splits ++on `char::is_whitespace`. They agree on every corpus here — the probe counts ++non-ASCII whitespace and measures **0** across both — but they would disagree ++on a non-breaking space or an en-quad. The clean fix needs char-boundary ++tracking across token boundaries, since a BPE token can split a multi-byte ++character; until a corpus that exercises it exists, the divergence is bounded ++by that count rather than closed. ++ ++**G8 — no table parser in-tree.** Neither `calamine` nor `csv` appears in any of ++the nine checkouts. "Parse a table" is genuinely unanswered — and the answer is ++a parser, not a DataFrame, because parsing a table does not require a DataFrame ++to own cognition. ++ ++## 8. The disable table ++ ++Every gate below was verified RED under the named change and GREEN without it. ++A gate nobody has tried to break is not evidence. ++ ++| disable | gate that went red | ++|---|---| ++| index the raw text instead of the receipt handle | `T-TANTIVY-NOSRC` | ++| receipt loses `particle_count` (PAD-scan to lane end) | `T-RECON` | ++| `byte_len` off by one on one id | `T-OFFSET` | ++| contract id ignores the normalisation rule | `T-CONTRACT-RULE` | ++| lexical unit span end off by one | `T-DEEPNSM-SPAN` | ++| a class address enters the token path | `T-FENCE` | ++| `view()` stops checking the contract id | `T-CONTRACT-GATE` | ++| the projection re-encodes the source | `T-DEEPNSM` | ++| forward windows copy instead of borrowing | `T-FORWARD` | ++| drop the CRLF normalisation (the original bug) | `T-CORPUS` | ++| every lexical unit claims a single-token span | `T-DEEPNSM-CARD` | ++| the PoS-blind control keeps the real tags | `T-DEEPNSM-FSM` | ++| build the phrase from receipt 1, assert receipt 0 | `T-TANTIVY-PHRASE` | ++ ++The last four exist because an independent vacuity audit of the finished probe ++found five holes, and all five were real: ++ ++- `T-CORPUS` asserted byte counts and never the SPAN count — so the CRLF bug ++ that collapsed 300 paragraphs into one span would have re-passed it. ++- `T-DEEPNSM-CARD` asserted `q50 >= 1`, which is true of any lexical unit that ++ exists at all, while its message claimed the cardinality was *measured*. It ++ now asserts the two facts it actually claims: a maximum above 1, and a ++ non-zero count of tokens straddling a word boundary. ++- `T-DEEPNSM-FSM` asserted only that some triple came out — a statement about a ++ type signature, not about behaviour. It now runs a **PoS-blind control**: the ++ same word ids with every tag flattened to `Noun` must produce a DIFFERENT ++ triple count, which is what makes "the FSM consumes `(WordId, Pos)`" a ++ measured claim. ++- `T-TANTIVY-PHRASE` checked `stored.starts_with("rcpt:")`, which is ++ unconditionally true of every document in that index. It now asserts equality ++ with the receipt the phrase was taken from. ++- `project()` splits on `u8::is_ascii_whitespace` where DeepNSM splits on ++ `char::is_whitespace`. Real divergence, unexercised here — now BOUNDED by ++ measurement (G9) rather than by hope. ++ ++**Two of the disables were themselves wrong first, and that is the part worth ++keeping.** (a) A first attempt at the framing disable replaced the ++`token_count` trim with a PAD scan *inside the receipt's own particle range* and ++stayed green — correctly, because within a bounded run a scan for a RESERVED id ++is exact. The gate's prose had over-claimed; it now states both lawful framings ++and the disable targets the actual unlawful one. (b) A first attempt at the ++codebook disable removed the rule id from the digest and stayed green, because ++on mixed-case text the two rules train different tables anyway. The gate now ++trains both rules on an already-lowercase corpus, where the tables are ++identical, and additionally asserts they behave differently on mixed-case ++input. **A knob that does not bind on the fixture is not a disable, and a ++fixture's SHAPE is part of a test's coverage.** ++ ++A third instance was pure apparatus: an early disable batch reported "no ++failure" for six changes in a row because the probe binary path was wrong and ++nothing ran. A null result is a claim about the measurement apparatus until ++proven otherwise. ++ ++## 9. Next rungs, in order ++ ++1. **G6 — the paged vocabulary.** Does `hi:lo` as `(page, id)` extend the ++ alphabet without widening a lane, and what does compression do at 1 MB? ++ Until measured, no scale claim. ++2. **G1 — offsets at the OCR boundary.** One upstream change, small, and it is ++ what makes the seam usable on scanned documents rather than on text files. ++3. **G2 — the lawful lane.** `SoaEnvelope` or a tenant, with the receipt column ++ layout designed against the measured 30–54 % framing overhead. ++4. **A real forward arm.** The seam supplies the input; whether representation ++ A, B or C wins is untested and needs training, not assertion. ++5. **F10 — a structured-evidence corpus** to exercise the parallel typed path ++ and the shared span identity where the two meet. +-- +2.43.0 + + +From 4b1ec2a5b4d8f5543abb4c59ffd2428b37d7b602 Mon Sep 17 00:00:00 2001 +From: Claude +Date: Sun, 23 Aug 2026 23:31:15 +0000 +Subject: [PATCH 2/4] re-cut the seam onto ogar-doc-ir: the receipt mints + nothing +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +The first cut invented `source_id: u32` / `span_id: u32` — a second population +wearing the document layer's job. `ogar-doc-ir` already answers all three +questions a tokenization receipt asks, so the seam now reads them instead: + + WHICH document -> DocIr::content_sha256, interned once per document + WHICH span -> a Region, at (DocPage::number, Region::reading_order) + the span's text -> Region::text + +Three consequences. + +`content_sha256` is a PER-ACQUISITION dedup key, not a cross-retina identity — +the crate's own docs correct its plan's first sketch on exactly this point, and +for a tokenization receipt the per-acquisition reading is the right one: you +tokenize bytes, so different bytes are a different tokenization. Cross-retina +convergence is a facts question and is not this seam's business. + +Byte offsets are REGION-LOCAL, because the region owns its text. That retires +the "the OCR boundary supplies no byte offsets" gap this report carried as the +thing blocking real documents. What remains is far smaller: a sub-region span +needs a non-zero byte_from, which the receipt already has and no producer emits. + +The seam is source-agnostic for free — docir.rs has no line that knows which +retina produced the IR. + +New gates (41 total, 18 disable-runs): + T-DOCIR both corpora round-trip through the IR's own load gate; an + off-vocabulary region kind AND a doc.v2 version are both + refused, so "closed vocabulary" is a mechanism not a word + T-DOCIR-KEY every receipt resolves to the IR's address, checked against the + DocIr walked independently; re-interning the same hash returns + the same index (S-2 dedup at lane scope) + T-DOCIR-SPANS a container descends, a figure contributes nothing, and a + table's cells are NOT flattened into the token stream + +Two method notes. T-DOCIR-KEY's first version compared each receipt against the +SAME spans() call it was validating — an implementation checked against itself +— and stayed green when spans() was changed to renumber by position; it now +walks the IR independently, and the fixture's reading_order is 2i+1 rather than +the positional index so the two are distinguishable at all. The table-flattening +disable initially found nothing because neither text corpus contains a table or +a figure; that was a fixture-coverage gap, closed with a three-region page. + +Also corrected: `uniq` is the count of ids APPEARING in the lane, not the +vocabulary size. The trained table is full at 255/255 on Alice and 180/255 on +the KJV fixture, where the corpus rather than the cap set it. An earlier draft +quoted the first number as the second. + +Co-Authored-By: Claude Opus 5 +Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf +--- + Cargo.toml | 5 + + crates/paperless-token/Cargo.toml | 6 + + .../examples/probe_token_seam.rs | 397 +++++++++++++++--- + crates/paperless-token/src/docir.rs | 97 +++++ + crates/paperless-token/src/lane.rs | 55 ++- + crates/paperless-token/src/lib.rs | 1 + + docs/TOKEN-SEAM-ARCHITECTURE.md | 132 ++++-- + 7 files changed, 581 insertions(+), 112 deletions(-) + create mode 100644 crates/paperless-token/src/docir.rs + +diff --git a/Cargo.toml b/Cargo.toml +index 50f16e6..4908d38 100644 +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -56,6 +56,11 @@ repository = "https://github.com/AdaWorldAPI/paperless-rs" + [workspace.dependencies] + tesseract-ogar = { git = "https://github.com/AdaWorldAPI/tesseract-rs", rev = "b369a9758d642d1b7f3a02b3aca2f53497f74531" } + ogar-vocab = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a5db99dd3fa92a532ba060eb0838" } ++# The source-agnostic perceptual IR — the document layer's OWN identity and ++# span addressing. The token seam consumes it rather than minting a parallel ++# `source_id`/`span_id`. Default features only: `classview` is OFF, so this ++# edge stays serde-only and pulls no canon dependency. ++ogar-doc-ir = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a5db99dd3fa92a532ba060eb0838" } + lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } + # The semantic consumer of the token seam. Standalone crate (its own empty + # `[workspace]` table) inside the same checkout; only dep is the contract crate +diff --git a/crates/paperless-token/Cargo.toml b/crates/paperless-token/Cargo.toml +index 1bc624a..05b9cca 100644 +--- a/crates/paperless-token/Cargo.toml ++++ b/crates/paperless-token/Cargo.toml +@@ -8,6 +8,12 @@ license.workspace = true + repository.workspace = true + + [dependencies] ++# The document layer's IR: `DocIr` supplies the span population AND the ++# identity a receipt is keyed by. A span is a `Region`, addressed by ++# `(page.number, reading_order)`; `Region::text` is its own canonical text, so ++# a byte offset is region-local and this crate never invents a document id. ++ogar-doc-ir.workspace = true ++ + # The contract identity. Already a workspace dep for the S-2 convergence key. + sha2.workspace = true + +diff --git a/crates/paperless-token/examples/probe_token_seam.rs b/crates/paperless-token/examples/probe_token_seam.rs +index a72b778..2cd89a9 100644 +--- a/crates/paperless-token/examples/probe_token_seam.rs ++++ b/crates/paperless-token/examples/probe_token_seam.rs +@@ -43,11 +43,17 @@ use std::collections::HashMap; + use std::sync::Arc; + + use deepnsm_v2::{parse_to_spo, PaletteVocab, Pos, Spo, Tagged}; ++use ogar_doc_ir::{ ++ from_json, to_json, BBoxRail, DocIr, DocPage, Geometry, Provenance, Rail, Region, RegionKind, ++ TableCell, DOC_IR_VERSION, ++}; + use paperless_token::contract::{query_passes, source_passes, NormRule, TokenizerContract}; ++use paperless_token::docir::{spans, SpanKey}; + use paperless_token::forward::{score, windows, CountPredictor}; + use paperless_token::lane::{TokenLane, TokenStreamReceipt, IDS_PER_PARTICLE}; + use paperless_token::lexical::project; + use paperless_token::seam_tantivy::{handle, ReceiptTokenizer, SeamStore, TermMode}; ++use sha2::{Digest, Sha256}; + use tantivy::collector::TopDocs; + use tantivy::query::PhraseQuery; + use tantivy::schema::{IndexRecordOption, Schema, TextFieldIndexing, TextOptions, Value}; +@@ -77,31 +83,75 @@ const SCENE: &[&str] = &[ + and I hid myself.", + ]; + +-/// One corpus under test. ++/// One corpus under test — carried as the document layer's own IR, not as a ++/// bag of strings this crate invented a span numbering for. + struct Corpus { + name: &'static str, +- /// The canonical text, and the span boundaries within it. +- text: String, +- spans: Vec<(u32, u32)>, ++ /// The ORIGINAL bytes, whose sha256 is the document identity. ++ source: Vec, ++ /// The perceptual IR a retina would have produced for those bytes. ++ ir: DocIr, + } + +-fn kjv() -> Corpus { +- let mut text = String::new(); +- let mut spans = Vec::new(); +- for v in SCENE { +- let from = u32::try_from(text.len()).expect("fits"); +- text.push_str(v); +- text.push('\n'); +- spans.push((from, u32::try_from(text.len()).expect("fits"))); ++/// Build a `DocIr` whose regions are the given paragraphs, in reading order. ++/// ++/// [`Geometry::DomOrder`] is the honest value: these are reading-order ++/// placements quantized onto the unit square, NOT measured layout. The IR has ++/// a variant for exactly that distinction and using `Rendered` here would ++/// claim a measurement nobody took. ++fn ir_from_paragraphs(source: &[u8], paras: &[String], prov: Provenance) -> DocIr { ++ let n = paras.len().max(1); ++ let regions = paras ++ .iter() ++ .enumerate() ++ .map(|(i, p)| { ++ let y0 = u8::try_from(i * 255 / n).unwrap_or(u8::MAX); ++ let y1 = u8::try_from((i + 1) * 255 / n).unwrap_or(u8::MAX); ++ Region { ++ kind: RegionKind::Text, ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: y0 }, ++ br: Rail { x: 255, y: y1 }, ++ }, ++ // Deliberately NOT the positional index: `2i+1` makes ++ // "read the field" and "renumber by position" distinguishable. ++ // With them equal, a gate that renumbers passes identically — ++ // the fixture's SHAPE is part of the coverage. ++ reading_order: u16::try_from(i * 2 + 1).unwrap_or(u16::MAX), ++ text: Some(p.clone()), ++ cells: Vec::new(), ++ children: Vec::new(), ++ } ++ }) ++ .collect(); ++ DocIr { ++ version: DOC_IR_VERSION.to_string(), ++ source: prov, ++ geometry: Geometry::DomOrder, ++ content_sha256: Sha256::digest(source).into(), ++ mime: "text/plain".to_string(), ++ pages: vec![DocPage { ++ number: 0, ++ width: 1, ++ height: u32::try_from(n).unwrap_or(u32::MAX), ++ regions, ++ }], ++ fields: Vec::new(), + } ++} ++ ++fn kjv() -> Corpus { ++ let paras: Vec = SCENE.iter().map(|v| (*v).to_string()).collect(); ++ let source = paras.join("\n").into_bytes(); ++ let ir = ir_from_paragraphs(&source, ¶s, Provenance::Ocr); + Corpus { + name: "kjv-genesis-scene", +- text, +- spans, ++ source, ++ ir, + } + } + +-/// Alice, split into paragraphs. Blank-line separated; a paragraph is the span. ++/// Alice, split into paragraphs. Blank-line separated; a paragraph is a region. + fn alice(max_spans: usize) -> Corpus { + // The committed file is CRLF with a BOM. A naive `split("\n\n")` finds + // NOTHING in it — the first version of this probe silently produced ONE +@@ -111,25 +161,23 @@ fn alice(max_spans: usize) -> Corpus { + let raw = include_str!("../corpus/alice.txt") + .trim_start_matches('\u{feff}') + .replace("\r\n", "\n"); +- let mut text = String::new(); +- let mut spans = Vec::new(); ++ let mut paras: Vec = Vec::new(); + for para in raw.split("\n\n") { + let p = para.split_whitespace().collect::>().join(" "); + if p.len() < 40 { + continue; + } +- let from = u32::try_from(text.len()).expect("fits"); +- text.push_str(&p); +- text.push('\n'); +- spans.push((from, u32::try_from(text.len()).expect("fits"))); +- if spans.len() >= max_spans { ++ paras.push(p); ++ if paras.len() >= max_spans { + break; + } + } ++ let source = paras.join("\n").into_bytes(); ++ let ir = ir_from_paragraphs(&source, ¶s, Provenance::Dom); + Corpus { + name: "alice-paragraphs", +- text, +- spans, ++ source, ++ ir, + } + } + +@@ -227,17 +275,20 @@ struct Summary { + /// Tokenize a whole corpus into ONE lane under ONE contract, then measure it. + fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, Summary) { + let mut lane = TokenLane::new(); ++ let doc = lane.intern_document(c.ir.content_sha256); ++ let sp = spans(&c.ir, doc); + let mut probes = 0usize; + let mut per_span: Vec = Vec::new(); + let before = source_passes(); +- for (i, &(from, to)) in c.spans.iter().enumerate() { +- let bytes = &c.text.as_bytes()[from as usize..to as usize]; ++ for s in &sp { + let (tokens, p) = contract +- .try_encode(bytes) ++ .try_encode(s.text.as_bytes()) + .expect("contract trained on this corpus"); + probes += p; + per_span.push(tokens.len().div_ceil(IDS_PER_PARTICLE)); +- lane.append(0, u32::try_from(i).expect("fits"), from, contract, &tokens); ++ // byte_from is 0: a whole region, and the offset is REGION-LOCAL ++ // because the region owns its canonical text. ++ lane.append(s.key, 0, contract, &tokens); + } + let passes = source_passes() - before; + g.run( +@@ -245,11 +296,57 @@ fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, + "T-PASSES[{}] exactly one source tokenization per span", + c.name + ), +- passes == c.spans.len(), ++ passes == sp.len(), + &format!( + "{passes} source tokenizations for {} spans (1.000 per span); the three consumers \ + below add ZERO further source passes — every one of them reads the lane", +- c.spans.len() ++ sp.len() ++ ), ++ ); ++ ++ // ---- the receipt is keyed by the DOCUMENT LAYER's address ---- ++ // Independent ground truth: walk the IR directly rather than trusting the ++ // same `spans()` call under test. ++ let truth: Vec<(u16, u16)> = ++ c.ir.pages ++ .iter() ++ .flat_map(|pg| pg.regions.iter().map(move |r| (pg.number, r.reading_order))) ++ .collect(); ++ let key_ok = truth.len() == lane.receipts().len() ++ && lane ++ .receipts() ++ .iter() ++ .zip(&truth) ++ .all(|(r, &(page, ro))| { ++ r.key.page == page ++ && r.key.reading_order == ro ++ && lane.document_of(r) == Some(&c.ir.content_sha256) ++ }) ++ // and the orders are genuinely not the positional index, so the check ++ // above cannot be satisfied by renumbering ++ && truth.iter().enumerate().any(|(i, &(_, ro))| ro as usize != i); ++ let reinterned = { ++ let mut l2 = lane.clone(); ++ let same = l2.intern_document(c.ir.content_sha256); ++ let other = l2.intern_document([0xAB; 32]); ++ same == doc && other != doc && l2.document_len() == 2 ++ }; ++ g.run( ++ &format!( ++ "T-DOCIR-KEY[{}] the receipt carries no id this crate minted", ++ c.name ++ ), ++ key_ok && reinterned && !sp.is_empty(), ++ &format!( ++ "every one of {} receipts resolves to the IR's own address — `content_sha256` for \ ++ WHICH document and `(page, reading_order)` for WHICH span, the reading order \ ++ `ogar-doc-ir` documents as the one the temporal stream and DeepNSM consume. \ ++ Re-interning the same `content_sha256` returns the SAME index and a different one \ ++ does not: that is the S-2 dedup property at lane scope. The hash is interned once \ ++ per document, not stamped on every receipt — at these span sizes a receipt is \ ++ already a third of the resident bytes and 32 more per span would have more than \ ++ doubled that for no addressing gain", ++ sp.len() + ), + ); + +@@ -257,11 +354,11 @@ fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, + let mut recon_ok = true; + let mut tokens_total = 0usize; + let mut uniq = std::collections::HashSet::new(); +- for (r, &(from, to)) in lane.receipts().iter().zip(&c.spans) { ++ for (r, sr) in lane.receipts().iter().zip(&sp) { + let v = lane.view(r, contract).expect("same contract"); + tokens_total += v.len(); + uniq.extend(v.ids().iter().copied()); +- if v.decode() != c.text.as_bytes()[from as usize..to as usize] { ++ if v.decode() != sr.text.as_bytes() { + recon_ok = false; + } + } +@@ -277,17 +374,17 @@ fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, + "{} spans, {tokens_total} tokens, {} distinct ids; decode reads the lane and the \ + codebook and nothing else — the canonical text stays authoritative and is never \ + consulted to read a span back", +- c.spans.len(), ++ sp.len(), + uniq.len() + ), + ); + + // ---- derived offsets ---- + let mut off_ok = true; +- for (r, &(from, _)) in lane.receipts().iter().zip(&c.spans) { ++ for (r, sr) in lane.receipts().iter().zip(&sp) { + let v = lane.view(r, contract).expect("same contract"); + // Ground truth computed the expensive way: decode each prefix. +- let mut cursor = from; ++ let mut cursor = 0u32; + for t in v.tokens() { + let truth_len = contract.decode(&[t.id]).0.len(); + if t.byte_from != cursor +@@ -297,7 +394,7 @@ fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, + } + cursor = t.byte_to; + } +- if cursor != c.spans[r.span_id as usize].1 { ++ if cursor as usize != sr.text.len() { + off_ok = false; + } + } +@@ -317,8 +414,8 @@ fn ingest(c: &Corpus, contract: &TokenizerContract, g: &mut Gate) -> (TokenLane, + + let summary = Summary { + name: c.name, +- bytes: c.text.len(), +- spans: c.spans.len(), ++ bytes: c.source.len(), ++ spans: sp.len(), + tokens: tokens_total, + uniq_tokens: uniq.len(), + particles: lane.particle_len(), +@@ -356,7 +453,8 @@ fn deepnsm_arm( + let mut flattened: Vec = Vec::new(); + let mut offsets_ok = true; + +- for r in lane.receipts() { ++ let sp = spans(&c.ir, 0); ++ for (r, sr) in lane.receipts().iter().zip(&sp) { + let v = lane.view(r, contract).expect("same contract"); + let lex = project(&v); + // A token that carries more than one unit's start is a token straddling +@@ -370,7 +468,7 @@ fn deepnsm_arm( + // The unit's byte span must address the canonical text and land on + // exactly the bytes it claims (modulo the normalisation that drops + // non-alphabetic characters inside a word). +- let raw = &c.text.as_bytes()[u.byte_from as usize..u.byte_to as usize]; ++ let raw = &sr.text.as_bytes()[u.byte_from as usize..u.byte_to as usize]; + let renorm: String = raw + .iter() + .filter(|b| b.is_ascii_alphabetic()) +@@ -671,7 +769,7 @@ fn main() { + let exotic_ws: usize = corpora + .iter() + .map(|c| { +- c.text ++ String::from_utf8_lossy(&c.source) + .chars() + .filter(|ch| ch.is_whitespace() && !ch.is_ascii_whitespace()) + .count() +@@ -680,9 +778,9 @@ fn main() { + g.run( + "T-CORPUS committed, real, hermetic; span population and the unicode-whitespace \ + divergence both measured", +- corpora.iter().all(|c| c.text.len() > 1000) +- && corpora[0].spans.len() >= 8 +- && corpora[1].spans.len() >= 100 ++ corpora.iter().all(|c| c.source.len() > 1000) ++ && spans(&corpora[0].ir, 0).len() >= 8 ++ && spans(&corpora[1].ir, 0).len() >= 100 + && vocab.len() > 10_000 + && exotic_ws == 0, + &format!( +@@ -697,21 +795,161 @@ fn main() { + char-level `is_whitespace`, occurs {exotic_ws} times across both corpora — the \ + divergence is bounded by measurement, and named as a gap rather than denied", + corpora[0].name, +- corpora[0].text.len(), +- corpora[0].spans.len(), ++ corpora[0].source.len(), ++ spans(&corpora[0].ir, 0).len(), + corpora[1].name, +- corpora[1].text.len(), +- corpora[1].spans.len(), ++ corpora[1].source.len(), ++ spans(&corpora[1].ir, 0).len(), + vocab.len() + ), + ); + ++ // ---- T-DOCIR: through the IR's OWN closed-vocabulary gate ---- ++ let mut roundtrip_ok = true; ++ let mut sha_ok = true; ++ for c in &corpora { ++ let json = to_json(&c.ir).expect("serialize"); ++ match from_json(&json) { ++ Ok(back) => { ++ if back != c.ir { ++ roundtrip_ok = false; ++ } ++ } ++ Err(_) => roundtrip_ok = false, ++ } ++ if c.ir.content_sha256 != <[u8; 32]>::from(Sha256::digest(&c.source)) { ++ sha_ok = false; ++ } ++ } ++ // The gate must also REFUSE: swap a region kind for one outside the closed ++ // vocabulary and the IR's own loader has to reject it, or "closed" is a ++ // word rather than a mechanism. Same for a version bump. ++ let off_vocab = to_json(&corpora[0].ir) ++ .expect("serialize") ++ .replace("\"kind\":\"text\"", "\"kind\":\"paragraph\""); ++ let refused = from_json(&off_vocab).is_err(); ++ let wrong_version = to_json(&corpora[0].ir) ++ .expect("serialize") ++ .replace("\"version\":\"doc.v1\"", "\"version\":\"doc.v2\""); ++ let version_refused = from_json(&wrong_version).is_err(); ++ g.run( ++ "T-DOCIR the span population comes from ogar-doc-ir, through its own load gate", ++ roundtrip_ok && sha_ok && refused && version_refused, ++ "both corpora round-trip `to_json` -> `from_json` unchanged and their \ ++ `content_sha256` is the sha256 of the ORIGINAL bytes; an off-vocabulary region kind \ ++ (`paragraph`) and a `doc.v2` version are BOTH refused by the IR's loader, so the \ ++ closed vocabulary is a mechanism and not a word. Note what that hash IS, per the \ ++ crate's own correction of its plan: a PER-ACQUISITION dedup key, not a cross-retina \ ++ identity — a scan and an HTML page of one invoice have different bytes. For a \ ++ TOKENIZATION receipt that is exactly the right reading: you tokenize bytes, so \ ++ different bytes are a different tokenization, and cross-retina convergence is a facts \ ++ question (`converges_on_facts`) that is not this seam's business. Geometry is \ ++ `DomOrder` on both corpora because these are reading-order placements; claiming \ ++ `Rendered` would assert a measurement nobody took", ++ ); ++ ++ // ---- T-DOCIR-SPANS: which regions become spans, and which do NOT ---- ++ // The two text corpora contain only `Text` regions, so nothing in them can ++ // falsify how a figure or a table is handled. This purpose-built IR can: ++ // a `Figure` has no text and a `Table` carries typed `(row, col)` cells ++ // that must NOT be poured into a token stream — flattening a table into ++ // text is the mistake the ingestion doctrine names, and pouring cells in ++ // here would destroy exactly the typed structure the structured path ++ // consumes. ++ let mixed = DocIr { ++ version: DOC_IR_VERSION.to_string(), ++ source: Provenance::Ocr, ++ geometry: Geometry::Rendered, ++ content_sha256: [7u8; 32], ++ mime: "image/png".to_string(), ++ pages: vec![DocPage { ++ number: 3, ++ width: 1000, ++ height: 2000, ++ regions: vec![ ++ Region { ++ kind: RegionKind::Main, ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: 0 }, ++ br: Rail { x: 255, y: 255 }, ++ }, ++ reading_order: 11, ++ text: None, // a pure container ++ cells: Vec::new(), ++ children: vec![Region { ++ kind: RegionKind::Text, ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: 0 }, ++ br: Rail { x: 255, y: 60 }, ++ }, ++ reading_order: 12, ++ text: Some("the nested paragraph".to_string()), ++ cells: Vec::new(), ++ children: Vec::new(), ++ }], ++ }, ++ Region { ++ kind: RegionKind::Figure, ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: 60 }, ++ br: Rail { x: 255, y: 120 }, ++ }, ++ reading_order: 13, ++ text: None, ++ cells: Vec::new(), ++ children: Vec::new(), ++ }, ++ Region { ++ kind: RegionKind::Table, ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: 120 }, ++ br: Rail { x: 255, y: 255 }, ++ }, ++ reading_order: 14, ++ text: None, ++ cells: vec![TableCell { ++ row: 0, ++ col: 0, ++ text: "Haemoglobin".to_string(), ++ bbox: BBoxRail { ++ tl: Rail { x: 0, y: 120 }, ++ br: Rail { x: 80, y: 140 }, ++ }, ++ confidence: 97, ++ }], ++ children: Vec::new(), ++ }, ++ ], ++ }], ++ fields: Vec::new(), ++ }; ++ let mixed_spans = spans(&mixed, 0); ++ let cell_text_leaked = mixed_spans.iter().any(|s| s.text.contains("Haemoglobin")); ++ g.run( ++ "T-DOCIR-SPANS a container descends, a figure contributes nothing, a table is not flattened", ++ mixed_spans.len() == 1 ++ && mixed_spans[0].text == "the nested paragraph" ++ && mixed_spans[0].key.reading_order == 12 ++ && mixed_spans[0].key.page == 3 ++ && !cell_text_leaked, ++ &format!( ++ "a 3-region page (a text-less `Main` container holding one `Text` child, a \ ++ `Figure`, and a `Table` with one cell) yields exactly {} span — the nested \ ++ paragraph, keyed (page 3, reading_order 12). The figure adds nothing, the \ ++ container adds nothing of its own, and the cell text \"Haemoglobin\" does NOT \ ++ appear in any span: a table's typed (row, col) values go to the structured path, \ ++ and pouring them into a token stream would destroy the structure that path \ ++ exists to read", ++ mixed_spans.len() ++ ), ++ ); ++ + let mut summaries: Vec = Vec::new(); + let mut index_rows: Vec<(String, IndexStats)> = Vec::new(); + let mut forward_rows: Vec<(String, usize, f64, usize, f64)> = Vec::new(); + + for c in &corpora { +- let contract = TokenizerContract::train(c.text.as_bytes(), NormRule::Identity); ++ let contract = TokenizerContract::train(&c.source, NormRule::Identity); + let (lane, mut s) = ingest(c, &contract, &mut g); + deepnsm_arm(c, &contract, &lane, &vocab, &posmap, &mut s, &mut g); + +@@ -810,11 +1048,11 @@ fn main() { + + // ---- T-CONTRACT: the codebook law ---- + let c0 = &corpora[0]; +- let a = TokenizerContract::train(c0.text.as_bytes(), NormRule::Identity); +- let b = TokenizerContract::train(c0.text.as_bytes(), NormRule::Identity); +- let other = TokenizerContract::train(corpora[1].text.as_bytes(), NormRule::Identity); +- let lower = TokenizerContract::train(c0.text.as_bytes(), NormRule::AsciiLowercase); +- let sample = c0.text.as_bytes(); ++ let a = TokenizerContract::train(&c0.source, NormRule::Identity); ++ let b = TokenizerContract::train(&c0.source, NormRule::Identity); ++ let other = TokenizerContract::train(&corpora[1].source, NormRule::Identity); ++ let lower = TokenizerContract::train(&c0.source, NormRule::AsciiLowercase); ++ let sample: &[u8] = &c0.source; + let (ta, _) = a.try_encode(sample).expect("trained on it"); + let (tb, _) = b.try_encode(sample).expect("trained on it"); + g.run( +@@ -853,9 +1091,9 @@ fn main() { + // an id while behaving differently on mixed-case input. That is the only + // corpus shape on which this claim is falsifiable; a mixed-case corpus + // makes the tables differ and the assertion passes for the wrong reason. +- let lc = c0.text.to_ascii_lowercase(); +- let r_id = TokenizerContract::train(lc.as_bytes(), NormRule::Identity); +- let r_lo = TokenizerContract::train(lc.as_bytes(), NormRule::AsciiLowercase); ++ let lc = c0.source.to_ascii_lowercase(); ++ let r_id = TokenizerContract::train(&lc, NormRule::Identity); ++ let r_lo = TokenizerContract::train(&lc, NormRule::AsciiLowercase); + let mixed = b"The Garden"; + let id_takes_mixed = r_id.try_encode(mixed).is_some(); + let lo_takes_mixed = r_lo.try_encode(mixed).is_some(); +@@ -879,10 +1117,28 @@ fn main() { + + // ---- T-FRAME: token_count is authoritative, PAD is not a length ---- + let mut frame_lane = TokenLane::new(); +- let (full, _) = a.try_encode(c0.text.as_bytes()).expect("trained"); ++ let (full, _) = a.try_encode(&c0.source).expect("trained"); + let exact = full.len() - (full.len() % IDS_PER_PARTICLE); // a 12-aligned run +- let r0 = frame_lane.append(0, 0, 0, &a, &full[..exact]); +- let r1 = frame_lane.append(0, 1, 0, &a, &full[exact..]); ++ let r0 = frame_lane.append( ++ SpanKey { ++ doc: 0, ++ page: 0, ++ reading_order: 0, ++ }, ++ 0, ++ &a, ++ &full[..exact], ++ ); ++ let r1 = frame_lane.append( ++ SpanKey { ++ doc: 0, ++ page: 0, ++ reading_order: 1, ++ }, ++ 0, ++ &a, ++ &full[exact..], ++ ); + let flat = frame_lane.particles().as_flattened(); + let pad_scan = flat + .iter() +@@ -914,13 +1170,13 @@ fn main() { + joined.extend_from_slice(&v1.decode()); + g.run( + "T-FRAME-ADJ adjacent receipts decode independently and concatenate exactly", +- v0.len() == exact && v1.len() == full.len() - exact && joined == c0.text.as_bytes(), ++ v0.len() == exact && v1.len() == full.len() - exact && joined == c0.source, + &format!( + "receipt 0 -> {} tokens, receipt 1 -> {} tokens, and their decodes concatenate back to \ + the full {} canonical bytes with no bleed in either direction", + v0.len(), + v1.len(), +- c0.text.len() ++ c0.source.len() + ), + ); + g.run( +@@ -1080,12 +1336,15 @@ fn main() { + println!( + "\nverdict: ONE receipt drove all three consumers. Tantivy indexed a receipt HANDLE and \ + never received the source; DeepNSM-v2 projected from ids alone through an unmodified \ +- library; the forward surface is a borrowed slice of the same particles. Byte offsets are \ +- DERIVED from the codebook's length table, so the receipt stores none. What is NOT settled \ +- here is stated in the report: the resident carrier is still a probe-local Vec, the OCR \ +- boundary supplies no byte offsets to attach a span to, the 8-bit vocabulary lane \ +- saturated at 75 KB, and there is no callable PoS surface anywhere — the module that \ +- held one was deliberately deleted, and the grounding cited for that deletion is \ +- itself an example binary outside this repo's dependency barrier." ++ library; the forward surface is a borrowed slice of the same particles. Byte offsets \ ++ are DERIVED from the codebook's length table, so the receipt stores none — and the \ ++ span population and every span's identity come from `ogar-doc-ir`, so the receipt \ ++ mints nothing either. An offset is therefore REGION-LOCAL, which is what retires the \ ++ old no-offsets-at-the-OCR-boundary gap. What is NOT settled: the resident carrier is \ ++ still a probe-local Vec; the 8-bit vocabulary table is FULL at 255/255 on 75 KB of \ ++ English; this probe builds its DocIr from text rather than from a real retina; and \ ++ there is no callable PoS surface anywhere — the module that held one was deliberately \ ++ deleted, and the grounding cited for that deletion is itself an example binary \ ++ outside this repo's dependency barrier." + ); + } +diff --git a/crates/paperless-token/src/docir.rs b/crates/paperless-token/src/docir.rs +new file mode 100644 +index 0000000..b6353e6 +--- /dev/null ++++ b/crates/paperless-token/src/docir.rs +@@ -0,0 +1,97 @@ ++//! The span population, taken from the document layer's own IR. ++//! ++//! ```text ++//! THE DOCUMENT LAYER ALREADY HAS AN IDENTITY. DO NOT MINT A SECOND ONE. ++//! ``` ++//! ++//! A tokenization receipt needs three things from upstream: WHICH document, ++//! WHICH span of it, and the span's canonical text. All three already exist in ++//! [`ogar_doc_ir::DocIr`], and an earlier cut of this crate invented parallel ++//! `source_id`/`span_id` integers instead of reading them — a second ++//! population wearing the document layer's job. ++//! ++//! - **which document** — [`ogar_doc_ir::DocIr::content_sha256`], the sha256 of ++//! the ORIGINAL bytes. The crate's own docs correct the plan's first sketch ++//! here and the correction matters for this seam: that hash is a ++//! **per-acquisition dedup key**, NOT a cross-retina semantic identity (a ++//! scan and an HTML page of the same invoice have different bytes). For a ++//! TOKENIZATION receipt the per-acquisition reading is exactly the right ++//! one: you tokenize bytes, and different bytes are a different tokenization. ++//! Cross-retina convergence is a facts question ++//! (`ogar_doc_ir::converges_on_facts`) and is not this seam's business. ++//! - **which span** — a [`ogar_doc_ir::Region`], addressed by its page number ++//! and its `reading_order`, which the IR documents as "the reading-order the ++//! temporal stream (and `DeepNSM`) consumes". The seam does not choose an ++//! order; it inherits the one the document layer already fixed. ++//! - **the text** — [`ogar_doc_ir::Region::text`]. Each region owns its own ++//! canonical text, so a byte offset is REGION-LOCAL. This is what dissolves ++//! the "the OCR boundary supplies no byte offsets" gap: the boundary supplies ++//! no PAGE-wide offsets, and does not need to. ++//! ++//! Because the IR is source-agnostic, so is the seam: a crawled page ++//! (`Provenance::Dom`) and a scan (`Provenance::Ocr`) present the same span ++//! population to the tokenizer, and this module contains not one line that ++//! knows which retina produced it. ++ ++use ogar_doc_ir::{DocIr, Region}; ++ ++/// Where a span sits in the document layer's own address space. ++/// ++/// The document half is an INDEX into [`crate::lane::TokenLane`]'s document ++/// table rather than a repeated 32-byte hash: at these span sizes a receipt is ++/// already a third of the resident bytes, and stamping the sha256 on every one ++/// of them would more than double that for no addressing gain. ++#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] ++pub struct SpanKey { ++ /// Index into the lane's document table (which holds the `content_sha256`). ++ pub doc: u16, ++ /// [`ogar_doc_ir::DocPage::number`]. ++ pub page: u16, ++ /// [`ogar_doc_ir::Region::reading_order`]. ++ pub reading_order: u16, ++} ++ ++/// One span the tokenizer will consume: its address and its canonical text. ++#[derive(Clone, Copy, Debug)] ++pub struct SpanRef<'a> { ++ /// The document-layer address. ++ pub key: SpanKey, ++ /// The region's own canonical text — the bytes that get tokenized. ++ pub text: &'a str, ++} ++ ++/// Walk a [`DocIr`] into its text-bearing spans, in page then reading order. ++/// ++/// Container regions are descended into; a region with no `text` contributes ++/// nothing itself (a `Figure` has no text, a `Main` holding tables is a ++/// container). Table cells are deliberately NOT flattened into text here — ++/// a cell is typed, addressed `(row, col)` data that the structured path ++/// consumes directly, and pouring it into a token stream is the ++/// flatten-the-table mistake the ingestion doctrine names. ++#[must_use] ++pub fn spans(ir: &DocIr, doc: u16) -> Vec> { ++ let mut out = Vec::new(); ++ for page in &ir.pages { ++ for r in &page.regions { ++ walk(r, doc, page.number, &mut out); ++ } ++ } ++ out ++} ++ ++fn walk<'a>(r: &'a Region, doc: u16, page: u16, out: &mut Vec>) { ++ // (Edition 2021 here, so no let-chain: the workspace pins edition 2021.) ++ if let Some(t) = r.text.as_deref().filter(|t| !t.is_empty()) { ++ out.push(SpanRef { ++ key: SpanKey { ++ doc, ++ page, ++ reading_order: r.reading_order, ++ }, ++ text: t, ++ }); ++ } ++ for c in &r.children { ++ walk(c, doc, page, out); ++ } ++} +diff --git a/crates/paperless-token/src/lane.rs b/crates/paperless-token/src/lane.rs +index f49ef41..b66a413 100644 +--- a/crates/paperless-token/src/lane.rs ++++ b/crates/paperless-token/src/lane.rs +@@ -30,6 +30,7 @@ + //! straight into the next receipt. The probe exercises exactly that case. + + use crate::contract::{TokenizerContract, PAD}; ++use crate::docir::SpanKey; + + /// Ids per particle: the 12-byte payload, one `u8` per byte. + pub const IDS_PER_PARTICLE: usize = 12; +@@ -41,25 +42,20 @@ pub const IDS_PER_PARTICLE_U32: u32 = 12; + /// The resident particle: the V3 content-blind 12-byte payload. + pub type TokenParticle = [u8; IDS_PER_PARTICLE]; + +-/// A source document's identity. Opaque to this crate — supplied by intake. +-pub type SourceId = u32; +- +-/// A span within a source (a page, a paragraph, a verse, a cell). +-pub type SpanId = u32; +- + /// What one tokenization produced, and everything needed to read it back. + /// + /// This is the RECEIPT. It carries no bytes and no offsets: the ids live in the + /// lane, and offsets are a prefix sum over the contract's per-id length table. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + pub struct TokenStreamReceipt { +- /// Which document. +- pub source_id: SourceId, +- /// Which span of it. +- pub span_id: SpanId, ++ /// WHERE, in the document layer's own address space — never an id this ++ /// crate minted. See [`crate::docir`]. ++ pub key: SpanKey, + /// Which codebook assigned these ids. Without it they are meaningless. + pub tokenizer_contract_id: [u8; 32], +- /// Byte offset of the span within the source's canonical text. ++ /// Byte offset of the span within its REGION's canonical text. A whole ++ /// region is 0; the field exists because a sub-region span is lawful and ++ /// would not be. + pub byte_from: u32, + /// AUTHORITATIVE token count. Not derivable from padding. + pub token_count: u32, +@@ -86,6 +82,9 @@ impl TokenStreamReceipt { + pub struct TokenLane { + particles: Vec, + receipts: Vec, ++ /// `content_sha256` per document, interned once. A receipt carries a ++ /// `u16` index into this, not the hash itself. ++ docs: Vec<[u8; 32]>, + } + + impl TokenLane { +@@ -95,6 +94,33 @@ impl TokenLane { + Self::default() + } + ++ /// Intern a document's `content_sha256`, returning the index a ++ /// [`SpanKey`] addresses it by. Re-interning the same hash returns the ++ /// same index — which IS the S-2 dedup property, at lane scope: the same ++ /// bytes acquired twice are one document here, not two. ++ /// ++ /// # Panics ++ /// If a lane accumulates more than `u16::MAX` documents. ++ pub fn intern_document(&mut self, content_sha256: [u8; 32]) -> u16 { ++ if let Some(i) = self.docs.iter().position(|d| *d == content_sha256) { ++ return u16::try_from(i).expect("bounded by the check below"); ++ } ++ self.docs.push(content_sha256); ++ u16::try_from(self.docs.len() - 1).expect("lane holds <= u16::MAX documents") ++ } ++ ++ /// The `content_sha256` a receipt's key addresses. ++ #[must_use] ++ pub fn document_of(&self, r: &TokenStreamReceipt) -> Option<&[u8; 32]> { ++ self.docs.get(r.key.doc as usize) ++ } ++ ++ /// Documents interned in this lane. ++ #[must_use] ++ pub fn document_len(&self) -> usize { ++ self.docs.len() ++ } ++ + /// Append one tokenized span. The ids are packed 12 per particle with a PAD + /// tail; `token_count` is recorded because the tail is not a length. + /// +@@ -104,8 +130,7 @@ impl TokenLane { + /// mis-framed span rather than a large one. + pub fn append( + &mut self, +- source_id: SourceId, +- span_id: SpanId, ++ key: SpanKey, + byte_from: u32, + contract: &TokenizerContract, + tokens: &[u8], +@@ -118,8 +143,7 @@ impl TokenLane { + } + let token_count = u32::try_from(tokens.len()).expect("span fits u32"); + let receipt = TokenStreamReceipt { +- source_id, +- span_id, ++ key, + tokenizer_contract_id: contract.contract_id(), + byte_from, + token_count, +@@ -156,6 +180,7 @@ impl TokenLane { + pub fn resident_bytes(&self) -> usize { + self.particles.len() * IDS_PER_PARTICLE + + self.receipts.len() * core::mem::size_of::() ++ + self.docs.len() * 32 + } + + /// A BORROWED view of one receipt's ids. No copy, no allocation: this is a +diff --git a/crates/paperless-token/src/lib.rs b/crates/paperless-token/src/lib.rs +index bfb8091..c2faf22 100644 +--- a/crates/paperless-token/src/lib.rs ++++ b/crates/paperless-token/src/lib.rs +@@ -33,6 +33,7 @@ + #![forbid(unsafe_code)] + + pub mod contract; ++pub mod docir; + pub mod forward; + pub mod lane; + pub mod lexical; +diff --git a/docs/TOKEN-SEAM-ARCHITECTURE.md b/docs/TOKEN-SEAM-ARCHITECTURE.md +index 5964c33..05a41dc 100644 +--- a/docs/TOKEN-SEAM-ARCHITECTURE.md ++++ b/docs/TOKEN-SEAM-ARCHITECTURE.md +@@ -27,8 +27,13 @@ load-bearing: the OCR boundary hands over no byte offsets to attach a span to + ## 1. The shape + + ``` +- canonical text (AUTHORITATIVE) ++ retina: tesseract-rs (pixels) │ spider-rs (DOM) + │ ++ ▼ ++ ogar_doc_ir::DocIr (AUTHORITATIVE) ++ content_sha256 · pages · regions · reading_order ++ │ ++ │ a span IS a Region; its text is Region::text + │ exactly one pass per span + ▼ + versioned BPE contract +@@ -57,6 +62,7 @@ shared table. + + | layer | owns | never | + |---|---|---| ++| `docir` | the span population, read from `ogar-doc-ir` | mints an identity | + | `contract` | the codebook and its identity | reads the lane | + | `lane` | resident particles + framing | owns text | + | `lexical` | the DeepNSM projection | reads the source | +@@ -66,6 +72,47 @@ shared table. + Each "never" is proven by a gate, not asserted — §8 lists which disable turns + each one red. + ++## 1b. The identity is the document layer's, not this crate's ++ ++An earlier cut of this crate minted `source_id: u32` and `span_id: u32`. That ++was a second population wearing the document layer's job, and `ogar-doc-ir` ++already answers all three questions a receipt has to ask: ++ ++| the receipt needs | `ogar-doc-ir` supplies | ++|---|---| ++| WHICH document | `DocIr::content_sha256` — sha256 of the ORIGINAL bytes | ++| WHICH span | a `Region`, addressed by `(DocPage::number, Region::reading_order)` | ++| the span's text | `Region::text` — each region owns its own canonical text | ++ ++Three consequences, and the second is the one that removes a gap this report ++previously carried: ++ ++- **`content_sha256` is a PER-ACQUISITION dedup key, not a cross-retina ++ identity.** The crate's own docs correct its plan's first sketch on exactly ++ this point: a scan and an HTML page of the same invoice have different bytes ++ and therefore different hashes. For a *tokenization* receipt that is the ++ right reading — you tokenize bytes, so different bytes are a different ++ tokenization. Cross-retina convergence is a facts question ++ (`converges_on_facts`) and is not this seam's business. ++- **Byte offsets are REGION-LOCAL, so the "no offsets at the OCR boundary" gap ++ largely dissolves.** The boundary supplies no PAGE-wide offset and does not ++ need to: the region owns its text, and `ogar-from-docv1::region_text` is ++ where tesseract's `leading_space`-aware join already happens. ++- **The seam is source-agnostic for free.** `docir.rs` contains no line that ++ knows which retina produced the IR, so a crawled page presents the same span ++ population as a scan. ++ ++The receipt interns the 32-byte hash **once per document** and carries a `u16` ++index. At these span sizes a receipt is already about a third of the resident ++bytes; stamping the hash on every one would have more than doubled that for no ++addressing gain. ++ ++Two things `docir::spans` deliberately does NOT do, both gated: a `Figure` ++contributes nothing (it has no text), and a `Table`'s cells are **not** ++flattened into the token stream — a cell is typed `(row, col)` data the ++structured path consumes, and pouring it into text destroys exactly the ++structure that path exists to read. ++ + ## 2. What was measured + + Two committed, real, hermetic corpora. `kjv-genesis-scene` is the same in-tree +@@ -77,15 +124,24 @@ Adventures in Wonderland*, carried from `tantivy/benches/alice.txt`. + + | corpus | bytes | spans | tokens | ratio | uniq ids | particles | resident B | particles/span p50/p95/max | continuation | + |---|---|---|---|---|---|---|---|---|---| +-| kjv-genesis-scene | 1 126 | 8 | 354 | 3.18× | 137 | 32 | 832 | 4 / 8 / 8 | 87.5 % | +-| alice-paragraphs | 75 514 | 300 | 37 149 | 2.03× | 247 | 3 231 | 55 572 | 8 / 30 / 43 | 100 % | ++| kjv-genesis-scene | 1 125 | 8 | 352 | 3.20× | 135 | 32 | 864 | 4 / 8 / 8 | 87.5 % | ++| alice-paragraphs | 75 513 | 300 | 36 994 | 2.04× | 245 | 3 214 | 55 400 | 8 / 30 / 43 | 100 % | + + Encode cost, in #1012's honest unit (merge-table probes, never wall time): +-86 640 and 7 929 358. +- +-**The resident lane is ~74 % of the source text, not a fraction of it.** For +-Alice the particles alone are 38 772 B (51 % of source) and the receipts add +-16 800 B. At these span sizes the FRAMING, not the payload, is where the bytes ++85 748 and 7 879 322. ++ ++The `uniq` column is the number of distinct ids that APPEAR in the lane, and ++that is not the vocabulary size — a distinction worth stating because an ++earlier draft of this report conflated them. The trained table is **full at ++255 of 255** on Alice (and still full on the whole 170 KB file); 245 of those ++ids occur. On the KJV fixture the table is **180 of 255** — there the CORPUS, ++not the cap, set the size, which is what #1016's own record of the fixture ++says. The two corpora sit on opposite sides of that line, and that is the ++interesting fact rather than either number alone. ++ ++**The resident lane is ~73 % of the source text, not a fraction of it.** For ++Alice the particles alone are 38 568 B (51 % of source), the 300 receipts add ++16 800 B, and the interned document hash adds 32. At these span sizes the FRAMING, not the payload, is where the bytes + go: a 56-byte receipt against 12-byte particles is 30 % of the resident total + at paragraph granularity and 54 % at verse granularity. #1012 could not see + this — it had no receipt. +@@ -243,9 +299,12 @@ fixture-scale `[u8;12]` result?** Five things, all now specified and gated: + truth. + - a **per-id surface table**, which is what lets every downstream projection + run without the source. +-- **span identity** `(source_id, span_id, byte_from)` — and the measured +- consequence that at these span sizes the receipt is 30–54 % of resident +- bytes, so the receipt's own layout matters more than the particle's. ++- **span identity — taken from `ogar-doc-ir`, not minted**: `content_sha256` ++ interned once per document, plus `(page, reading_order)` per span, plus a ++ region-local `byte_from`. The measured consequence is that at these span ++ sizes the receipt is ~30 % of resident bytes, so the receipt's own layout ++ matters more than the particle's — which is also why the hash is interned ++ rather than stamped per span. + + **5. Does any online step still require Polars?** No — and the honest form of + that answer is that it never did. A sweep of nine checkouts found **zero** +@@ -266,10 +325,11 @@ DataFrame (G8). + + **6. What remains authoritative?** In order, and the order is enforced rather + than described: +-`canonical source text` (authoritative) → `contract` (authoritative for what an +-id MEANS) → `receipt + lane` (exact and reconstructible; byte-identical decode +-on both corpora) → `Tantivy index` (derived, deletable) → `DeepNSM projection` +-(derived, deletable) → `model state` (ephemeral, owns nothing). ++`ogar_doc_ir::DocIr` (authoritative — it owns the document's identity, its span ++population and each span's canonical text) → `contract` (authoritative for what ++an id MEANS) → `receipt + lane` (exact and reconstructible; byte-identical ++decode on both corpora) → `Tantivy index` (derived, deletable) → `DeepNSM ++projection` (derived, deletable) → `model state` (ephemeral, owns nothing). + + **7. Can Tantivy be deleted and rebuilt without semantic loss?** Yes. It holds + terms and positions derived from the lane, stores a handle rather than text, +@@ -360,15 +420,15 @@ decision needs and did not have. + + ## 7. The named gaps + +-**G1 — the OCR boundary supplies no byte offsets, and this is the one that +-blocks real documents.** `doc.v1` carries `bbox`, `conf`, `leading_space` and a +-page-level `plain_text`, and no `offset`/`span`/`start`/`end` field on any of +-`DocWord`/`DocLine`/`DocPage`. The word→text mapping is *derivable* by replaying +-the documented join rule, but nothing stores it — so intake would have to +-re-implement a rule that lives in `tesseract-ocr/src/renderer.rs`, which is a +-second implementation of one rule and will drift. The fix belongs upstream and +-is small: emit the per-word byte offset where the join is performed, since that +-is the one place the rule exists. ++**G1 — RESOLVED by reading `ogar-doc-ir`; what remains is much smaller.** This ++report previously carried "the OCR boundary supplies no byte offsets" as the ++gap that blocked real documents. It is not one. `doc.v1` carries no page-wide ++offset, but the perceptual IR does not need one: a span is a `Region` and ++`Region::text` is its own canonical text, so an offset is region-local and ++`ogar-from-docv1::region_text` is where the `leading_space`-aware join already ++lives. What actually remains: a SUB-region span (half a paragraph) needs a ++non-zero `byte_from`, which the receipt already carries and no producer yet ++emits. + + **G2 — no resident SoA carrier.** The lane is a probe-local `Vec`. Lawful + options: implement `SoaEnvelope`, or mint a new `ValueTenant` (16 variants +@@ -445,6 +505,11 @@ A gate nobody has tried to break is not evidence. + | every lexical unit claims a single-token span | `T-DEEPNSM-CARD` | + | the PoS-blind control keeps the real tags | `T-DEEPNSM-FSM` | + | build the phrase from receipt 1, assert receipt 0 | `T-TANTIVY-PHRASE` | ++| `spans()` renumbers by position instead of reading `reading_order` | `T-DOCIR-KEY` | ++| `intern_document` never dedups | `T-DOCIR-KEY` | ++| the IR hashes something other than the source bytes | `T-DOCIR` | ++| `spans()` flattens table cells into the token stream | `T-DOCIR-SPANS` | ++| a text-less container emits an empty span | `T-DOCIR-SPANS` | + + The last four exist because an independent vacuity audit of the finished probe + found five holes, and all five were real: +@@ -467,8 +532,17 @@ found five holes, and all five were real: + `char::is_whitespace`. Real divergence, unexercised here — now BOUNDED by + measurement (G9) rather than by hope. + +-**Two of the disables were themselves wrong first, and that is the part worth +-keeping.** (a) A first attempt at the framing disable replaced the ++**Three of the disables were themselves wrong first, and that is the part worth ++keeping.** The third is from the `ogar-doc-ir` re-cut: the first version of ++`T-DOCIR-KEY` compared each receipt's key against the SAME `spans()` call it ++was validating — an implementation checked against itself, which stayed green ++when `spans()` was changed to renumber by position. It now walks the `DocIr` ++independently, and the fixture's `reading_order` is deliberately `2i+1` rather ++than the positional index, so "reads the field" and "renumbers by position" are ++distinguishable at all. A fourth disable (flattening table cells) initially ++found nothing for a different reason — neither text corpus contains a table or ++a figure, so no fixture could express it. That is a coverage gap, not a sound ++gate, and it was closed by building a three-region page that has both. (a) A first attempt at the framing disable replaced the + `token_count` trim with a PAD scan *inside the receipt's own particle range* and + stayed green — correctly, because within a bounded run a scan for a RESERVED id + is exact. The gate's prose had over-claimed; it now states both lawful framings +@@ -490,8 +564,10 @@ proven otherwise. + 1. **G6 — the paged vocabulary.** Does `hi:lo` as `(page, id)` extend the + alphabet without widening a lane, and what does compression do at 1 MB? + Until measured, no scale claim. +-2. **G1 — offsets at the OCR boundary.** One upstream change, small, and it is +- what makes the seam usable on scanned documents rather than on text files. ++2. **A real retina.** This probe builds its `DocIr` from text; the next one ++ should take `ogar-from-docv1`'s output on an actual scan and a ++ `spider_doc_ir` crawl of the same content, and check that both present the ++ same span population shape to the tokenizer. + 3. **G2 — the lawful lane.** `SoaEnvelope` or a tenant, with the receipt column + layout designed against the measured 30–54 % framing overhead. + 4. **A real forward arm.** The seam supplies the input; whether representation +-- +2.43.0 + + +From 070b93560795cc96c1782d100ac37b0dad58d098 Mon Sep 17 00:00:00 2001 +From: Claude +Date: Sun, 23 Aug 2026 23:57:17 +0000 +Subject: [PATCH 3/4] two measurement probes: carrier width, and WordId vs + TokenId in the tile +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Neither changes library code. Both are examples; all three probes still run and +the seam probe is still 41/41. fmt + clippy -D warnings clean. + +PROBE-TILE-H2H-1 — flat u8 (255-cap, 12 per particle) vs the 256:256 rail token +(u16, one per rail, 6 per particle). Same trainer, same corpora, same spans; +only cap and packing differ. On alice-full (166 KB, 805 spans): + + u8 : table CAP-bound at 255, ratio 1.95x, resident 87.6 KB + u16: table 7 675 of 65 535 CORPUS-bound (88% of the tile free), + ratio 6.13x, resident 56.8 KB (-35%), 3.1x fewer tokens + +So the earlier "saturates at 75 KB" was a property of a cap the first cut +invented, not of the corpus. kjv-scene lands identical under both caps (table +198, corpus-bound below 255) — the sanity check that only the cap differs. +H2H-SHAPE verifies a rail token is the same split/join arithmetic as +deepnsm_v2::vocab on a u16, in a SEPARATE tile (nothing touches PaletteVocab). + +The trainer is a positions-list BPE (linked list + lazy heap); the naive +full-pass form carried from #1012 is O(merges x stream) and unusable at a +65 536 cap. Both widths use it, so the comparison is fair, and every span is +validated by byte-exact decode rather than assumption. + +PROBE-WORDID-VS-TOKENID-1 — what each candidate puts INTO the tile, same corpus: + + carrier units distinct of its space reconstruct hi byte means + TokenId u8/255 85241 244 95.31% EXACT merge order + WordId (DeepNSM) 28104 1858 2.84% 67.7% freq band + TokenId u16 rail 27134 6351 9.69% EXACT merge order + WordNet synset - 65292 99.63% n/a ancestry + +Findings. (1) The rail token and WordId land on near-identical unit counts +(27 134 vs 28 104) — at rail width a BPE token is about word-sized on this +corpus, so they do not compete on granularity. (2) WordId recovers only 67.7% +of source bytes; 32.3% is gone to case, punctuation, digits and sub-2-letter +words. Not a defect — it is a semantic coordinate, never a codec — but it +settles the carrier question: a WordId stream cannot be canonical. (3) Only +WordNet puts meaning in the ADDRESS. TokenId's hi byte is merge order; WordId's +is a frequency band; DeepNSM's actual meaning lives in the cam96 codebook, +which is a separate structure from the id (and ABSENT here). + +WordNet's row is CITED, not re-measured: wordnetcode.princeton.edu is outside +this environment's egress allowlist. Figures come from lance-graph's +PROBE-WORDNET-44-ACTIVATION (5/5) with provenance at the print site. Worth +noting: 65 292 noun leaves against a 65 536 tile is a 99.6% fit. + +Co-Authored-By: Claude Opus 5 +Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf +--- + .../examples/probe_tile_head_to_head.rs | 455 ++++++++++++++++++ + .../examples/probe_wordid_vs_tokenid.rs | 241 ++++++++++ + 2 files changed, 696 insertions(+) + create mode 100644 crates/paperless-token/examples/probe_tile_head_to_head.rs + create mode 100644 crates/paperless-token/examples/probe_wordid_vs_tokenid.rs + +diff --git a/crates/paperless-token/examples/probe_tile_head_to_head.rs b/crates/paperless-token/examples/probe_tile_head_to_head.rs +new file mode 100644 +index 0000000..0aebbcc +--- /dev/null ++++ b/crates/paperless-token/examples/probe_tile_head_to_head.rs +@@ -0,0 +1,455 @@ ++//! `PROBE-TILE-H2H-1` — head-to-head: the flat 8-bit token id (255-cap, the ++//! first cut's invention) vs the **256:256 rail token** (`u16`, 65 536-cap, ++//! one token per `(hi:lo)` rail, six per 12-byte particle — the native V3 ++//! reading, and byte-for-byte the same SHAPE as `DeepNSM`-v2's `WordId`). ++//! ++//! Separate tiles, same shape (operator ruling): a token id lives in its OWN ++//! 256×256 tile; `WordId` lives in `DeepNSM`'s. Nothing here touches ++//! `PaletteVocab` — the shape is shared, the space is not. ++//! ++//! Same trainer, same corpora, same span population. ONLY the cap and the ++//! packing differ, so any difference in the table is the carrier's. ++//! ++//! # Honesty box ++//! - The 8-bit side's "saturation at 75 KB" was an artifact of a cap the first ++//! cut invented. This probe measures what the substrate's own width does on ++//! the same bytes. ++//! - Cost is reported as merge/apply operation counts, never wall time. ++//! - The trainer here is a positions-list BPE (linked token list + lazy heap), ++//! NOT the naive full-pass one carried from #1012 — the naive form is ++//! O(merges × stream) and unusable at a 65 536 cap. Both widths use THIS ++//! trainer, so the comparison stays fair; its output is validated by ++//! byte-exact reconstruction, not assumed. ++ ++#![allow( ++ clippy::many_single_char_names, ++ clippy::too_many_lines, ++ clippy::cast_precision_loss ++)] ++ ++use std::collections::{BinaryHeap, HashMap}; ++ ++/// Dead-node marker in the linked token list. ++const DEAD: u32 = u32::MAX; ++ ++#[derive(Clone, Copy)] ++enum Exp { ++ Base(u8), ++ Pair(u32, u32), ++} ++ ++/// One trained table, id width = u32 internally, capped by `cap` (255 for the ++/// u8 carrier, `65_535` for the u16 rail carrier — one id reserved as PAD in ++/// each width). ++struct Bpe { ++ expand: Vec, ++ base_of: HashMap, ++ /// (left,right) -> merged id, in training order (rank = id). ++ ranks: HashMap<(u32, u32), u32>, ++ strings: Vec>, ++ hit_cap: bool, ++ apply_ops: usize, ++} ++ ++impl Bpe { ++ fn train(corpus: &[u8], cap: usize) -> Self { ++ // Base alphabet. ++ let mut base_of: HashMap = HashMap::new(); ++ let mut expand: Vec = Vec::new(); ++ let mut strings: Vec> = Vec::new(); ++ for &b in corpus { ++ base_of.entry(b).or_insert_with(|| { ++ expand.push(Exp::Base(b)); ++ strings.push(vec![b]); ++ u32::try_from(expand.len() - 1).expect("alphabet fits") ++ }); ++ } ++ // Linked token list. ++ let n = corpus.len(); ++ let mut id: Vec = corpus.iter().map(|b| base_of[b]).collect(); ++ let n32 = u32::try_from(n).expect("corpus fits u32"); ++ let mut next: Vec = (1..=n32).collect(); ++ let mut prev: Vec = (0..n32).map(|i| i.wrapping_sub(1)).collect(); ++ if n > 0 { ++ next[n - 1] = DEAD; ++ } ++ // Pair counts, occurrence positions (left-node index), lazy max-heap. ++ let mut counts: HashMap<(u32, u32), i64> = HashMap::new(); ++ let mut occ: HashMap<(u32, u32), Vec> = HashMap::new(); ++ let mut heap: BinaryHeap<(i64, (u32, u32))> = BinaryHeap::new(); ++ for i in 0..n.saturating_sub(1) { ++ let p = (id[i], id[i + 1]); ++ *counts.entry(p).or_default() += 1; ++ occ.entry(p) ++ .or_default() ++ .push(u32::try_from(i).expect("fits")); ++ } ++ for (&p, &c) in &counts { ++ heap.push((c, p)); ++ } ++ let mut ranks: HashMap<(u32, u32), u32> = HashMap::new(); ++ let mut apply_ops = 0usize; ++ let mut hit_cap = false; ++ loop { ++ if expand.len() >= cap { ++ hit_cap = true; ++ break; ++ } ++ // Pop until a live entry. ++ let Some((c, p)) = heap.pop() else { break }; ++ if counts.get(&p).copied().unwrap_or(0) != c { ++ continue; // stale ++ } ++ if c < 2 { ++ break; ++ } ++ let new_id = u32::try_from(expand.len()).expect("bounded by cap"); ++ expand.push(Exp::Pair(p.0, p.1)); ++ let mut s = strings[p.0 as usize].clone(); ++ s.extend_from_slice(&strings[p.1 as usize]); ++ strings.push(s); ++ ranks.insert(p, new_id); ++ counts.insert(p, 0); ++ let positions = occ.remove(&p).unwrap_or_default(); ++ for i in positions { ++ apply_ops += 1; ++ let i = i as usize; ++ // Validate: node alive, still this pair. ++ if id[i] == DEAD || id[i] != p.0 { ++ continue; ++ } ++ let j = next[i]; ++ if j == DEAD || id[j as usize] != p.1 { ++ continue; ++ } ++ let j = j as usize; ++ // Merge j into i. ++ let l = prev[i]; ++ let r = next[j]; ++ id[i] = new_id; ++ id[j] = DEAD; ++ next[i] = r; ++ if r != DEAD { ++ prev[r as usize] = u32::try_from(i).expect("fits"); ++ } ++ // Left neighbour pair updates. ++ if l != DEAD { ++ let ol = (id[l as usize], p.0); ++ let e = counts.entry(ol).or_default(); ++ *e -= 1; ++ heap.push((*e, ol)); ++ let nl = (id[l as usize], new_id); ++ let e = counts.entry(nl).or_default(); ++ *e += 1; ++ heap.push((*e, nl)); ++ occ.entry(nl).or_default().push(l); ++ } ++ // Right neighbour pair updates. ++ if r != DEAD { ++ let or_ = (p.1, id[r as usize]); ++ let e = counts.entry(or_).or_default(); ++ *e -= 1; ++ heap.push((*e, or_)); ++ let nr = (new_id, id[r as usize]); ++ let e = counts.entry(nr).or_default(); ++ *e += 1; ++ heap.push((*e, nr)); ++ occ.entry(nr) ++ .or_default() ++ .push(u32::try_from(i).expect("fits")); ++ } ++ } ++ } ++ Self { ++ expand, ++ base_of, ++ ranks, ++ strings, ++ hit_cap, ++ apply_ops, ++ } ++ } ++ ++ /// Encode a span: repeatedly merge the lowest-rank adjacent pair present. ++ /// Returns `None` on a byte outside the trained alphabet. ++ fn encode(&self, src: &[u8], ops: &mut usize) -> Option> { ++ let mut s: Vec = Vec::with_capacity(src.len()); ++ for b in src { ++ s.push(*self.base_of.get(b)?); ++ } ++ loop { ++ let mut best: Option<(u32, usize)> = None; ++ for k in 0..s.len().saturating_sub(1) { ++ *ops += 1; ++ if let Some(&r) = self.ranks.get(&(s[k], s[k + 1])) { ++ if best.is_none_or(|(br, _)| r < br) { ++ best = Some((r, k)); ++ } ++ } ++ } ++ let Some((rank, _)) = best else { break }; ++ let (a, b) = match self.expand[rank as usize] { ++ Exp::Pair(a, b) => (a, b), ++ Exp::Base(_) => unreachable!("ranks map only holds pairs"), ++ }; ++ let mut out = Vec::with_capacity(s.len()); ++ let mut k = 0; ++ while k < s.len() { ++ if k + 1 < s.len() && s[k] == a && s[k + 1] == b { ++ out.push(rank); ++ k += 2; ++ } else { ++ out.push(s[k]); ++ k += 1; ++ } ++ } ++ s = out; ++ } ++ Some(s) ++ } ++ ++ fn decode(&self, tokens: &[u32]) -> Vec { ++ let mut out = Vec::new(); ++ let mut stack = Vec::new(); ++ for &t in tokens { ++ stack.push(t); ++ while let Some(x) = stack.pop() { ++ match self.expand[x as usize] { ++ Exp::Base(b) => out.push(b), ++ Exp::Pair(a, b) => { ++ stack.push(b); ++ stack.push(a); ++ } ++ } ++ } ++ } ++ out ++ } ++} ++ ++/// One measured row. ++struct Row { ++ corpus: &'static str, ++ width: &'static str, ++ cap: usize, ++ table: usize, ++ hit_cap: bool, ++ tokens: usize, ++ ratio: f64, ++ particles: usize, ++ particle_bytes: usize, ++ uniq: usize, ++ max_surface: usize, ++ train_ops: usize, ++ enc_ops: usize, ++} ++ ++fn paragraphs(raw: &str, max: usize) -> Vec { ++ let raw = raw.trim_start_matches('\u{feff}').replace("\r\n", "\n"); ++ let mut out = Vec::new(); ++ for para in raw.split("\n\n") { ++ let p = para.split_whitespace().collect::>().join(" "); ++ if p.len() < 40 { ++ continue; ++ } ++ out.push(p); ++ if out.len() >= max { ++ break; ++ } ++ } ++ out ++} ++ ++fn run(corpus: &'static str, spans: &[String], width: &'static str, cap: usize, per: usize) -> Row { ++ let joined = spans.join("\n").into_bytes(); ++ let t = Bpe::train(&joined, cap); ++ let mut tokens = 0usize; ++ let mut particles = 0usize; ++ let mut enc_ops = 0usize; ++ let mut uniq = std::collections::HashSet::new(); ++ for s in spans { ++ let ids = t ++ .encode(s.as_bytes(), &mut enc_ops) ++ .expect("trained on this corpus"); ++ assert_eq!( ++ t.decode(&ids), ++ s.as_bytes(), ++ "[FAIL] H2H-RECON {corpus}/{width}: span does not decode byte-exact" ++ ); ++ tokens += ids.len(); ++ particles += ids.len().div_ceil(per); ++ uniq.extend(ids.iter().copied()); ++ } ++ let src: usize = spans.iter().map(String::len).sum(); ++ Row { ++ corpus, ++ width, ++ cap, ++ table: t.expand.len(), ++ hit_cap: t.hit_cap, ++ tokens, ++ ratio: src as f64 / tokens as f64, ++ particles, ++ particle_bytes: particles * 12, ++ uniq: uniq.len(), ++ max_surface: uniq ++ .iter() ++ .map(|&i| t.strings[i as usize].len()) ++ .max() ++ .unwrap_or(0), ++ train_ops: t.apply_ops, ++ enc_ops, ++ } ++} ++ ++fn main() { ++ println!("PROBE-TILE-H2H-1 — flat u8 (255-cap) vs 256:256 rail token (u16, 65 536-cap)\n"); ++ ++ // ---- H2H-SHAPE: the rail token IS WordId's shape, in a separate tile ---- ++ // Same split/join arithmetic as deepnsm_v2::vocab — proven by calling both ++ // on a sweep, not by asserting prose. Separate tiles: nothing in this probe ++ // constructs or consults a PaletteVocab. ++ for id in [0u16, 1, 0x00FF, 0x0100, 0x50AA, 0xFFFE] { ++ let (hi, lo) = ((id >> 8) as u8, (id & 0xFF) as u8); ++ assert_eq!(deepnsm_v2::vocab::split(id), (hi, lo), "shape drift"); ++ assert_eq!(deepnsm_v2::vocab::join(hi, lo), id, "shape drift"); ++ } ++ println!( ++ " [PASS] H2H-SHAPE a rail token (hi:lo) and a DeepNSM WordId (basin:identity) are the \ ++ SAME split/join arithmetic on a u16 — verified against deepnsm_v2::vocab::split/join \ ++ on a sweep incl. both byte boundaries. Separate tiles: this probe never touches \ ++ PaletteVocab, so the shape is shared and the space is not" ++ ); ++ ++ let kjv: Vec = super_scene(); ++ let alice300 = paragraphs(include_str!("../corpus/alice.txt"), 300); ++ let alice_full = paragraphs(include_str!("../corpus/alice.txt"), usize::MAX); ++ println!( ++ "\n corpora: kjv-scene {} B / {} spans; alice-300 {} B / {} spans; alice-full {} B / {} spans\n", ++ kjv.iter().map(String::len).sum::(), ++ kjv.len(), ++ alice300.iter().map(String::len).sum::(), ++ alice300.len(), ++ alice_full.iter().map(String::len).sum::(), ++ alice_full.len() ++ ); ++ ++ let mut rows = Vec::new(); ++ for (name, spans) in [ ++ ("kjv-scene", &kjv), ++ ("alice-300", &alice300), ++ ("alice-full", &alice_full), ++ ] { ++ rows.push(run(name, spans, "u8/12per", 255, 12)); ++ rows.push(run(name, spans, "u16/6per", 65_535, 6)); ++ } ++ ++ println!( ++ "{:<12} {:<9} {:>6} {:>7} {:>8} {:>8} {:>7} {:>6} {:>10} {:>7} {:>6} {:>11} {:>11}", ++ "corpus", ++ "carrier", ++ "cap", ++ "table", ++ "capped?", ++ "tokens", ++ "ratio", ++ "uniq", ++ "maxB/tok", ++ "part.", ++ "res.KB", ++ "train ops", ++ "enc ops" ++ ); ++ for r in &rows { ++ println!( ++ "{:<12} {:<9} {:>6} {:>7} {:>8} {:>8} {:>7.2} {:>6} {:>10} {:>7} {:>6.1} {:>11} {:>11}", ++ r.corpus, ++ r.width, ++ r.cap, ++ r.table, ++ if r.hit_cap { "CAP" } else { "corpus" }, ++ r.tokens, ++ r.ratio, ++ r.uniq, ++ r.max_surface, ++ r.particles, ++ r.particle_bytes as f64 / 1024.0, ++ r.train_ops, ++ r.enc_ops ++ ); ++ } ++ ++ // ---- H2H-SAT: who set the table size, the cap or the corpus? ---- ++ let u8_alice = rows ++ .iter() ++ .find(|r| r.corpus == "alice-full" && r.width == "u8/12per") ++ .expect("row"); ++ let u16_alice = rows ++ .iter() ++ .find(|r| r.corpus == "alice-full" && r.width == "u16/6per") ++ .expect("row"); ++ assert!( ++ u8_alice.hit_cap, ++ "[FAIL] H2H-SAT: expected the u8 table to be cap-bound on alice-full" ++ ); ++ assert!( ++ !u16_alice.hit_cap, ++ "[FAIL] H2H-SAT: expected the u16 table to be corpus-bound on alice-full" ++ ); ++ println!( ++ "\n [PASS] H2H-SAT on alice-full the u8 table is CAP-bound at {} while the u16 table \ ++ stops at {} of 65 535 — the CORPUS set it (no adjacent pair repeats), with {:.1}% of \ ++ the tile still free. The old \"saturates at 75 KB\" was the cap's property, not the \ ++ corpus's", ++ u8_alice.table, ++ u16_alice.table, ++ 100.0 * (65_535.0 - u16_alice.table as f64) / 65_535.0 ++ ); ++ let bytes_u8 = u8_alice.particle_bytes; ++ let bytes_u16 = u16_alice.particle_bytes; ++ println!( ++ "\n resident particle bytes, alice-full: u8 {} vs u16 {} ({}) — six wider tokens per \ ++ particle vs twelve narrower ones; the RATIO of surface covered per particle is what \ ++ decides, and it is measured above, not assumed", ++ bytes_u8, ++ bytes_u16, ++ if bytes_u16 < bytes_u8 { ++ format!( ++ "u16 smaller by {:.1}%", ++ 100.0 * (bytes_u8 - bytes_u16) as f64 / bytes_u8 as f64 ++ ) ++ } else { ++ format!( ++ "u16 LARGER by {:.1}%", ++ 100.0 * (bytes_u16 - bytes_u8) as f64 / bytes_u8 as f64 ++ ) ++ } ++ ); ++ println!("\nPROBE-TILE-H2H-1: measured; reconstruction byte-exact on every span, both widths"); ++} ++ ++/// The same KJV scene the seam probe uses, as spans. ++fn super_scene() -> Vec { ++ [ ++ "But of the tree of the knowledge of good and evil, thou shalt not eat of it: for in \ ++ the day that thou eatest thereof thou shalt surely die.", ++ "And they were both naked, the man and his wife, and were not ashamed.", ++ "Now the serpent was more subtil than any beast of the field which the LORD God had \ ++ made. And he said unto the woman, Yea, hath God said, Ye shall not eat of every tree \ ++ of the garden?", ++ "And the serpent said unto the woman, Ye shall not surely die:", ++ "And when the woman saw that the tree was good for food, and that it was pleasant to \ ++ the eyes, and a tree to be desired to make one wise, she took of the fruit thereof, \ ++ and did eat, and gave also unto her husband with her; and he did eat.", ++ "And the eyes of them both were opened, and they knew that they were naked; and they \ ++ sewed fig leaves together, and made themselves aprons.", ++ "And they heard the voice of the LORD God walking in the garden in the cool of the \ ++ day: and Adam and his wife hid themselves from the presence of the LORD God amongst \ ++ the trees of the garden.", ++ "And he said, I heard thy voice in the garden, and I was afraid, because I was naked; \ ++ and I hid myself.", ++ ] ++ .iter() ++ .map(|s| (*s).to_string()) ++ .collect() ++} +diff --git a/crates/paperless-token/examples/probe_wordid_vs_tokenid.rs b/crates/paperless-token/examples/probe_wordid_vs_tokenid.rs +new file mode 100644 +index 0000000..8f07621 +--- /dev/null ++++ b/crates/paperless-token/examples/probe_wordid_vs_tokenid.rs +@@ -0,0 +1,241 @@ ++//! `PROBE-WORDID-VS-TOKENID-1` — side by side: what does each candidate put ++//! INTO the 256:256 tile, and what does it cost you? ++//! ++//! Three candidates want the same two bytes. They are not variants of one ++//! thing; they differ in what the ADDRESS means and in what you can get back. ++//! ++//! | | the hi byte means | reconstruction | ++//! |---|---|---| ++//! | `WordId` (`DeepNSM`-v2) | frequency band | LOSSY — measured below | ++//! | `TokenId` (BPE rail) | merge order (nothing) | EXACT — measured below | ++//! | `WordNet` synset | taxonomic ancestry | n/a — not a tokenization | ++//! ++//! Separate tiles, same shape (operator ruling). This probe measures the first ++//! two on identical spans. The third is CITED, not re-measured: `WordNet`'s ++//! source host is outside this environment's network egress allowlist ++//! (`Host not in allowlist: wordnetcode.princeton.edu`), so the corpus cannot ++//! be fetched here — but lance-graph already measured it ++//! (`PROBE-WORDNET-44-ACTIVATION`, 5/5) and those numbers are quoted with ++//! their provenance rather than re-derived. ++//! ++//! # Honesty box ++//! - `PaletteVocab` is loaded from the committed COCA projection; the trained ++//! `cam96` codebook is ABSENT, so `DeepNSM`'s SEMANTIC DISTANCE is not ++//! exercised. What is measured is the id space and the round trip. ++//! - The `WordId` round trip joins surviving words with single spaces. That is ++//! the most favourable reconstruction available — the normalisation genuinely ++//! discards case, punctuation and short words, so a kinder join cannot exist. ++//! - No claim is made about which is "better". They answer different questions. ++ ++// A probe is one linear argument: splitting `main` into helpers hides the order ++// the measurements run in, which is what a reader checks. ++#![allow( ++ clippy::many_single_char_names, ++ clippy::cast_precision_loss, ++ clippy::too_many_lines ++)] ++ ++use std::collections::{HashMap, HashSet}; ++ ++use deepnsm_v2::PaletteVocab; ++use paperless_token::contract::{NormRule, TokenizerContract}; ++ ++/// `WordNet`, quoted from `PROBE-WORDNET-44-ACTIVATION` (lance-graph, 5/5 gates). ++/// Not re-measured here: the corpus host is outside the egress allowlist. ++const WORDNET_SYNSETS: usize = 82_192; ++const WORDNET_LEAVES: usize = 65_292; ++const WORDNET_BAND_RECALL: f64 = 0.763; ++const WORDNET_RANDOM_RECALL: f64 = 0.031; ++ ++/// The 256:256 rail carrier on this same corpus, from `PROBE-TILE-H2H-1`. ++/// Cited, not re-trained here — see the note at the print site. ++const RAIL_UNITS: usize = 27_134; ++const RAIL_DISTINCT: usize = 6_351; ++ ++/// The `DeepNSM` lexical rule, from its own consumers. ++fn normalise(tok: &str) -> Option { ++ let w: String = tok ++ .chars() ++ .filter(char::is_ascii_alphabetic) ++ .collect::() ++ .to_lowercase(); ++ (w.len() >= 2).then_some(w) ++} ++ ++fn paragraphs(raw: &str, max: usize) -> Vec { ++ let raw = raw.trim_start_matches('\u{feff}').replace("\r\n", "\n"); ++ let mut out = Vec::new(); ++ for para in raw.split("\n\n") { ++ let p = para.split_whitespace().collect::>().join(" "); ++ if p.len() >= 40 { ++ out.push(p); ++ } ++ if out.len() >= max { ++ break; ++ } ++ } ++ out ++} ++ ++fn main() { ++ println!("PROBE-WORDID-VS-TOKENID-1 — what goes in the 256:256 tile\n"); ++ ++ // ---- vocabulary (committed COCA projection) ---- ++ let tsv = include_str!("../corpus/coca_academic_20k.tsv"); ++ let mut vocab = PaletteVocab::new(); ++ let words: Vec<&str> = tsv ++ .lines() ++ .filter_map(|l| l.split('\t').next()) ++ .filter(|w| !w.is_empty()) ++ .collect(); ++ vocab.from_frequency_ranked(words); ++ ++ let spans = paragraphs(include_str!("../corpus/alice.txt"), usize::MAX); ++ let src_bytes: usize = spans.iter().map(String::len).sum(); ++ ++ // ---- TokenId arm: BPE over the 256:256 rail tile ---- ++ let joined = spans.join("\n").into_bytes(); ++ let contract = TokenizerContract::train(&joined, NormRule::Identity); ++ let mut tok_units = 0usize; ++ let mut tok_ids: HashSet = HashSet::new(); ++ let mut tok_exact = true; ++ for s in &spans { ++ let (ids, _) = contract.try_encode(s.as_bytes()).expect("trained"); ++ if contract.decode(&ids).0 != s.as_bytes() { ++ tok_exact = false; ++ } ++ tok_units += ids.len(); ++ tok_ids.extend(ids.iter().copied()); ++ } ++ ++ // ---- WordId arm: the same spans through DeepNSM's own lexical rule ---- ++ let mut word_units = 0usize; ++ let mut word_resolved = 0usize; ++ let mut word_ids: HashSet = HashSet::new(); ++ let mut basins: HashSet = HashSet::new(); ++ let mut recovered_bytes = 0usize; ++ let mut oov: HashMap = HashMap::new(); ++ for s in &spans { ++ let mut rebuilt: Vec<&str> = Vec::new(); ++ for t in s.split_whitespace() { ++ let Some(w) = normalise(t) else { continue }; ++ word_units += 1; ++ if let Some(id) = vocab.id(&w) { ++ word_resolved += 1; ++ word_ids.insert(id); ++ basins.insert(deepnsm_v2::vocab::split(id).0); ++ rebuilt.push(vocab.word(id).expect("just resolved")); ++ } else { ++ *oov.entry(w).or_default() += 1; ++ } ++ } ++ // The kindest possible round trip: surviving words, single-spaced. ++ recovered_bytes += rebuilt.join(" ").len(); ++ } ++ ++ println!( ++ "corpus: alice-full, {} spans, {} source bytes\n", ++ spans.len(), ++ src_bytes ++ ); ++ println!( ++ "{:<16} {:>9} {:>9} {:>12} {:>13} {:>11}", ++ "carrier", "units", "distinct", "of its space", "reconstruct", "hi byte =", ++ ); ++ println!( ++ "{:<16} {:>9} {:>9} {:>11.2}% {:>13} {:>11}", ++ "TokenId u8/255", ++ tok_units, ++ tok_ids.len(), ++ 100.0 * tok_ids.len() as f64 / 256.0, ++ if tok_exact { "EXACT" } else { "BROKEN" }, ++ "merge order", ++ ); ++ println!( ++ "{:<16} {:>9} {:>9} {:>11.2}% {:>12.1}% {:>11}", ++ "WordId (DeepNSM)", ++ word_units, ++ word_ids.len(), ++ 100.0 * word_ids.len() as f64 / 65_536.0, ++ 100.0 * recovered_bytes as f64 / src_bytes as f64, ++ "freq band", ++ ); ++ // The rail carrier's row is CITED from PROBE-TILE-H2H-1 rather than ++ // re-trained here: this crate's `contract.rs` is still the u8 carrier, and ++ // scoring a u8 vocabulary against a 65 536 tile is the wrong-quantity ++ // mistake. Provenance is explicit so the cell cannot be read as measured ++ // here. ++ println!( ++ "{:<16} {:>9} {:>9} {:>11.2}% {:>13} {:>11}", ++ "TokenId u16 rail", ++ RAIL_UNITS, ++ RAIL_DISTINCT, ++ 100.0 * RAIL_DISTINCT as f64 / 65_536.0, ++ "EXACT", ++ "merge order", ++ ); ++ println!( ++ "{:<16} {:>9} {:>9} {:>11.2}% {:>13} {:>11}", ++ "WordNet synset", ++ "-", ++ WORDNET_LEAVES, ++ 100.0 * WORDNET_LEAVES as f64 / 65_536.0, ++ "n/a", ++ "ancestry", ++ ); ++ ++ let lost = src_bytes - recovered_bytes; ++ println!( ++ "\nWordId round trip: {} of {} source bytes recover ({:.1}%); {} bytes ({:.1}%) are \ ++ GONE — case, punctuation, digits and every sub-2-letter word. {} of {} lexical units \ ++ resolved ({:.1}% OOV). This is not a defect: DeepNSM's id is a SEMANTIC coordinate \ ++ and was never a codec. But it means a WordId stream cannot be the canonical carrier — \ ++ you cannot get the document back from it.", ++ recovered_bytes, ++ src_bytes, ++ 100.0 * recovered_bytes as f64 / src_bytes as f64, ++ lost, ++ 100.0 * lost as f64 / src_bytes as f64, ++ word_resolved, ++ word_units, ++ 100.0 * (word_units - word_resolved) as f64 / word_units as f64, ++ ); ++ println!( ++ "\nTokenId round trip: byte-exact on every span. At the 256:256 rail the tile is \ ++ {:.1}% EMPTY ({} ids used of 65 536) — the corpus, not the cap, set that. But its hi byte is merge \ ++ order: two ids sharing a basin share NOTHING semantic, so address adjacency in this \ ++ tile is not a search prior.", ++ 100.0 * (65_536.0 - RAIL_DISTINCT as f64) / 65_536.0, ++ RAIL_DISTINCT, ++ ); ++ println!( ++ "\nWordId basins actually occupied: {} of 256. The hi byte is the FREQUENCY band, so \ ++ basin adjacency means 'similar corpus frequency' — which is not semantic adjacency \ ++ either. DeepNSM gets its meaning from the trained cam96 codebook, which is ABSENT \ ++ here and is a SEPARATE structure from the id.", ++ basins.len(), ++ ); ++ println!( ++ "\nWordNet, CITED not re-measured (host outside this environment's egress allowlist; \ ++ lance-graph PROBE-WORDNET-44-ACTIVATION 5/5): {} noun synsets, {} leaves — {:.1}% of \ ++ a 65 536 tile, a near-exact fit. Folded to 4^4 = 256 cells, ancestry-band recall is \ ++ {:.3} vs {:.3} random = {:.1}x. That is the ONLY one of the three whose ADDRESS \ ++ carries meaning: prefix = ancestor, by construction rather than by training.", ++ WORDNET_SYNSETS, ++ WORDNET_LEAVES, ++ 100.0 * WORDNET_LEAVES as f64 / 65_536.0, ++ WORDNET_BAND_RECALL, ++ WORDNET_RANDOM_RECALL, ++ WORDNET_BAND_RECALL / WORDNET_RANDOM_RECALL, ++ ); ++ let mut top: Vec<(&String, &usize)> = oov.iter().collect(); ++ top.sort_by(|a, b| b.1.cmp(a.1).then(a.0.cmp(b.0))); ++ println!( ++ "\ntop OOV against the academic vocabulary: {}", ++ top.iter() ++ .take(12) ++ .map(|(w, c)| format!("{w}({c})")) ++ .collect::>() ++ .join(" ") ++ ); ++} +-- +2.43.0 + + +From 4a3db4e393781d75dc0253b0f4ab7680c103ea20 Mon Sep 17 00:00:00 2001 +From: Claude +Date: Mon, 24 Aug 2026 00:24:15 +0000 +Subject: [PATCH 4/4] wire both retinas into intake: hash before you spend, one + shape out +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +`paperless-intake` was a stub that existed only to exercise the [patch] +chain. It is now the S-2 gate in front of the two producers that already +emit `ogar_doc_ir::DocIr`, and it writes no producer of its own. + +Three entry points, each hashing BEFORE it touches a retina: + + ingest_html DOM spider_doc_ir::harvest (feature `dom`) + ingest_doc_v1 pixel ogar_from_docv1::from_doc_v1 (always) + ingest_image pixel OcrExecutor -> doc.v1 -> above (feature `ocr`) + +S-2 says dedup precedes recognition SPEND, and that is a per-entry-point +property — so each leg proves it separately, with an input the retina +itself would reject. `a_duplicate_never_reaches_the_doc_v1_seam` feeds +`}{ not json`; landing on Duplicate can only mean the gate answered +first. Disable-verified: moving the seam ahead of the short-circuit +turns it red, and the paired novel-input assertion stops it passing for +the wrong reason. + +Identity is READ, never minted. `spider_doc_ir::harvest` hashes +`content.as_bytes()`; `preflight` hashes the same bytes; the DOM test +asserts the two agree rather than assuming it. The pixel leg hashes the +IMAGE, not the JSON, so two renderings of one scan are one document. + +The `ocr` leg is off by default because the doc.v1 leg converts JSON +from any producer with no recognizer dependency at all — pulling +tesseract (and ndarray under it) into a consumer that ingests web pages +is a real cost for nothing. It compiles and lints clean when enabled. + +The `dom` leg is off for a worse reason, stated in full in the manifest: +`spider_doc_ir` does not compile against `ogar-doc-ir` main. It omits +`TableCell::confidence`, added upstream after spider's only commit, and +spider floats on `branch = "main"` — so it broke the moment the field +landed and nothing builds the pair to notice. One line fixes it +upstream; this session cannot push there. Nothing here works around it, +because working around a broken producer is how two IRs diverge. + +The workspace manifest floats the doc-IR pair on `main`, deviating from +its own no-branch-refs rule and saying so. cargo does not unify a +`branch` and a `rev` source even at the same commit (measured: two +ogar-doc-ir entries, identical #719471db), and two SourceIds means two +DocIr TYPES — which defeats a source-agnostic IR entirely. + +Co-Authored-By: Claude Opus 5 +Claude-Session: https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf +--- + Cargo.toml | 23 +- + crates/paperless-intake/Cargo.toml | 58 +++- + crates/paperless-intake/src/lib.rs | 411 ++++++++++++++++++++++++++++- + 3 files changed, 489 insertions(+), 3 deletions(-) + +diff --git a/Cargo.toml b/Cargo.toml +index 4908d38..f984218 100644 +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -60,7 +60,28 @@ ogar-vocab = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a + # span addressing. The token seam consumes it rather than minting a parallel + # `source_id`/`span_id`. Default features only: `classview` is OFF, so this + # edge stays serde-only and pulls no canon dependency. +-ogar-doc-ir = { git = "https://github.com/AdaWorldAPI/OGAR", rev = "70296bec67d1a5db99dd3fa92a532ba060eb0838" } ++# ⚠ DELIBERATE DEVIATION from this file's own "branch refs are forbidden" rule, ++# for the doc-IR pair ONLY. `spider_doc_ir` declares ++# `ogar-doc-ir = { branch = "main" }`, and cargo does NOT unify a `branch` and a ++# `rev` source even when both resolve to the SAME commit (measured: two ++# `ogar-doc-ir` entries in Cargo.lock, identical `#719471db` suffix). Two ++# SourceIds means two `DocIr` TYPES in one binary, so `spider_doc_ir::harvest`'s ++# output could not be handed to `ogar_from_docv1`'s consumer — which defeats the ++# entire point of a source-agnostic IR. ++# ++# Floating is the lesser evil here, and the drift protection is STRUCTURAL ++# rather than a pin: `ogar_doc_ir::from_json` hard-fails on an off-vocabulary ++# `RegionKind` or a `DOC_IR_VERSION` that is not `doc.v1`. That gate catches ++# SEMANTIC drift, which a rev pin cannot. spider floats for the same reason and ++# says so in its own manifest. ++ogar-doc-ir = { git = "https://github.com/AdaWorldAPI/OGAR", branch = "main" } ++# The PIXEL retina's adapter: tesseract's `doc.v1` JSON -> `DocIr`. serde-only, ++# so this edge costs nothing; note it does NOT pull tesseract itself — a caller ++# that already has doc.v1 from anywhere converts without an OCR dependency. ++ogar-from-docv1 = { git = "https://github.com/AdaWorldAPI/OGAR", branch = "main" } ++# The DOM retina: HTML -> `DocIr`, one function, `lol_html` + `sha2`. Merged in ++# the spider fork; a web page and a scan therefore land as ONE shape. ++spider_doc_ir = { git = "https://github.com/AdaWorldAPI/spider", branch = "main" } + lance-graph-contract = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "51c54faef235454401c6a7461ea05929c19f2140" } + # The semantic consumer of the token seam. Standalone crate (its own empty + # `[workspace]` table) inside the same checkout; only dep is the contract crate +diff --git a/crates/paperless-intake/Cargo.toml b/crates/paperless-intake/Cargo.toml +index 4527cbe..6c7132c 100644 +--- a/crates/paperless-intake/Cargo.toml ++++ b/crates/paperless-intake/Cargo.toml +@@ -9,4 +9,60 @@ repository.workspace = true + + [dependencies] + paperless-kv = { path = "../paperless-kv" } +-tesseract-ogar.workspace = true ++ ++# The one shape both retinas produce. ++ogar-doc-ir.workspace = true ++ ++# The PIXEL retina's adapter. serde-only, and it does NOT pull tesseract: a ++# caller holding doc.v1 from anywhere converts with no OCR dependency. ++ogar-from-docv1.workspace = true ++ ++# The DOM retina. OPTIONAL and OFF by default — not by preference but because ++# it does not currently compile: see the `dom` feature below. ++spider_doc_ir = { workspace = true, optional = true } ++ ++# Running the pixel retina IN-PROCESS is optional. Default OFF: the doc.v1 leg ++# converts JSON from any producer without an OCR dependency at all, and pulling ++# the recognizer (and ndarray under it) into every consumer that only needs to ++# ingest a web page would be a real cost for nothing. ++tesseract-ogar = { workspace = true, optional = true } ++ ++[features] ++default = [] ++ ++# Run the pixel retina IN-PROCESS. Off by default: pulling the recognizer (and ++# `ndarray` beneath it) into a consumer that only converts doc.v1 is a real ++# cost for nothing. ++ocr = ["dep:tesseract-ogar"] ++ ++# The DOM retina (`spider_doc_ir::harvest`). OFF because it is BROKEN UPSTREAM, ++# not because it is unwanted: ++# ++# error[E0063]: missing field `confidence` in initializer of `TableCell` ++# --> spider_doc_ir/src/lib.rs:145 ++# ++# `spider_doc_ir` was merged 2026-07-14 and has not been touched since; ++# `ogar-doc-ir` later added `TableCell::confidence` (a fail-closed `0` default, ++# so an OCR cell of 95 and 40 reports 40 and routes to review). spider declares ++# `ogar-doc-ir = { branch = "main" }`, so it broke the moment that field landed ++# — and nothing builds the pair, so nothing noticed. The fix is one line in ++# spider (`confidence: 0`, which is exactly the documented fail-closed default ++# for a producer that has no per-cell signal); this session cannot push there. ++# ++# The lesson is worth more than the fix: a floating branch dep with no CI that ++# exercises it is drift with NO detector. `ogar-doc-ir`'s closed vocabulary and ++# version marker guard the DATA shape at runtime; neither can see a struct ++# field added at compile time. ++dom = ["dep:spider_doc_ir"] ++ ++[dev-dependencies] ++# The tests construct a `DocumentGuid`, which wraps a `FacetCascade`. ++lance-graph-contract.workspace = true ++ ++[lints.rust] ++unsafe_code = "forbid" ++missing_docs = "warn" ++ ++[lints.clippy] ++all = { level = "deny", priority = -1 } ++pedantic = { level = "warn", priority = -1 } +diff --git a/crates/paperless-intake/src/lib.rs b/crates/paperless-intake/src/lib.rs +index 2feaf37..35153be 100644 +--- a/crates/paperless-intake/src/lib.rs ++++ b/crates/paperless-intake/src/lib.rs +@@ -1 +1,410 @@ +-//! Intake stub — exists to prove the [patch] wiring resolves. ++//! Intake: raw bytes → the S-2 gate → a retina → one `DocIr`. ++//! ++//! ```text ++//! HASH BEFORE YOU SPEND. TWO RETINAS, ONE SHAPE. ++//! ``` ++//! ++//! # The two retinas, and why neither is written here ++//! ++//! Both already exist and both already emit [`ogar_doc_ir::DocIr`]: ++//! ++//! | retina | producer | this crate calls | ++//! |---|---|---| ++//! | DOM (a web page) | `spider_doc_ir` (`AdaWorldAPI/spider`) | `ingest_html` (feature `dom`) | ++//! | pixel (a scan) | tesseract-rs → `doc.v1` → `ogar-from-docv1` | [`ingest_doc_v1`] | ++//! ++//! ⚠ The DOM leg is behind the OFF-by-default `dom` feature because ++//! `spider_doc_ir` does not currently compile against `ogar-doc-ir` main — it ++//! omits `TableCell::confidence`, a field added upstream after spider's only ++//! commit (2026-07-14). One line upstream fixes it; the manifest carries the ++//! detail. Nothing here works around it, because working around a broken ++//! producer is how two IRs start diverging. ++//! ++//! So intake's job is not to build a producer. It is to run the **gate** in ++//! front of them, hand back one shape, and be honest about which retina ++//! produced it — which [`DocIr::source`] already records ++//! ([`ogar_doc_ir::Provenance`]), so nothing here has to track it separately. ++//! ++//! # The pixel retina has two legs, and the split is deliberate ++//! ++//! [`ingest_doc_v1`] takes `doc.v1` JSON from ANY producer and needs no OCR ++//! dependency at all. [`ingest_image`] runs the recognizer in-process and is ++//! behind the **off-by-default `ocr` feature**, because pulling the recognizer ++//! (and `ndarray` beneath it) into a consumer that only ingests web pages is a ++//! real cost for nothing. ++//! ++//! # The gate runs FIRST, and that is the whole point of S-2 ++//! ++//! `OGAR-DOC-INGESTION-SPINE` S-2: dedup must precede recognition SPEND. Every ++//! entry point here hashes and asks the index *before* touching a retina — ++//! [`ingest_image`] especially, where the retina is the expensive one. A ++//! duplicate costs a hash and a lookup. The gate matches BOTH the original and ++//! derived-artifact hashes (S-2's second half), which [`paperless_kv::preflight`] ++//! already implements. ++//! ++//! # What `content_sha256` means, stated because the IR's own docs correct ++//! their plan on it ++//! ++//! It is the hash of the ORIGINAL bytes — a **per-acquisition dedup key**, not ++//! a cross-retina identity. The same invoice as a scan and as HTML has ++//! different bytes and therefore different hashes; cross-retina convergence is ++//! a facts question (`ogar_doc_ir::converges_on_facts`). Per-acquisition is ++//! exactly what a dedup gate wants, so the two agree here rather than merely ++//! coexisting: `spider_doc_ir::harvest` hashes `content.as_bytes()`, which is ++//! the same hash [`paperless_kv::preflight`] computes over the same bytes — ++//! asserted, not assumed (see `dom_identity_matches_the_gate`). ++ ++#![forbid(unsafe_code)] ++ ++use ogar_doc_ir::DocIr; ++use paperless_kv::{ContentSha256, DedupIndex, MatchedOn, Verdict}; ++ ++/// What one intake attempt produced. ++#[derive(Debug, Clone)] ++pub enum Ingested { ++ /// The gate matched. No retina ran. ++ Duplicate { ++ /// The hash that matched. ++ hash: ContentSha256, ++ /// Which stored hash it matched — original bytes, or a derived ++ /// artifact (S-2's second half). ++ matched: MatchedOn, ++ }, ++ /// Novel bytes; a retina ran and produced one shape. ++ Novel { ++ /// The per-acquisition dedup key. ++ hash: ContentSha256, ++ /// The perceptual IR. [`DocIr::source`] says which retina. ++ ir: Box, ++ }, ++} ++ ++impl Ingested { ++ /// The IR, if a retina ran. ++ #[must_use] ++ pub fn ir(&self) -> Option<&DocIr> { ++ match self { ++ Self::Novel { ir, .. } => Some(ir), ++ Self::Duplicate { .. } => None, ++ } ++ } ++ ++ /// The hash either way — a duplicate still has an identity. ++ #[must_use] ++ pub const fn hash(&self) -> &ContentSha256 { ++ match self { ++ Self::Novel { hash, .. } | Self::Duplicate { hash, .. } => hash, ++ } ++ } ++} ++ ++/// Why an intake attempt failed. ++#[derive(Debug)] ++pub enum IntakeError { ++ /// The DOM retina needs UTF-8; these bytes are not. ++ #[cfg(feature = "dom")] ++ NotUtf8(core::str::Utf8Error), ++ /// The pixel retina's adapter refused the JSON — malformed, wrong schema, ++ /// or a region kind outside the closed vocabulary. Fail-loud is the point: ++ /// a producer that drifts is caught at the seam. ++ DocV1(ogar_from_docv1::FromDocV1Error), ++ /// The recognizer refused the page (`ocr` feature only). ++ #[cfg(feature = "ocr")] ++ Ocr(tesseract_ogar::OcrExecError), ++ /// The recognizer answered, but not with a document (`ocr` feature only). ++ /// Structurally unreachable for a `RecognizeDocument` request; kept ++ /// because an enum match that cannot fail is a claim, not a guarantee. ++ #[cfg(feature = "ocr")] ++ NotADocument, ++} ++ ++impl core::fmt::Display for IntakeError { ++ fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { ++ match self { ++ #[cfg(feature = "dom")] ++ Self::NotUtf8(e) => write!(f, "the DOM retina needs UTF-8: {e}"), ++ Self::DocV1(e) => write!(f, "doc.v1 refused at the seam: {e}"), ++ #[cfg(feature = "ocr")] ++ Self::Ocr(e) => write!(f, "recognition failed: {e:?}"), ++ #[cfg(feature = "ocr")] ++ Self::NotADocument => { ++ write!(f, "the recognizer returned a non-document response") ++ } ++ } ++ } ++} ++ ++impl std::error::Error for IntakeError {} ++ ++/// DOM retina: an HTML page → one `DocIr`, gated. ++/// ++/// # Errors ++/// [`IntakeError::NotUtf8`] if the bytes are not UTF-8. ++#[cfg(feature = "dom")] ++pub fn ingest_html(bytes: &[u8], index: &I) -> Result { ++ let (hash, verdict) = paperless_kv::preflight(bytes, index); ++ if let Verdict::Duplicate { matched } = verdict { ++ return Ok(Ingested::Duplicate { hash, matched }); ++ } ++ let html = core::str::from_utf8(bytes).map_err(IntakeError::NotUtf8)?; ++ Ok(Ingested::Novel { ++ hash, ++ ir: Box::new(spider_doc_ir::harvest(html)), ++ }) ++} ++ ++/// Pixel retina, JSON leg: `doc.v1` from any producer → one `DocIr`, gated. ++/// ++/// `source_bytes` are the ORIGINAL image bytes — what the gate hashes and what ++/// the IR's identity is taken over. The JSON is a rendition of them, so ++/// hashing the JSON instead would make two renderings of one scan look like ++/// two documents. ++/// ++/// # Errors ++/// [`IntakeError::DocV1`] if the JSON is malformed, carries the wrong schema ++/// marker, or names a region kind outside the closed vocabulary. ++pub fn ingest_doc_v1( ++ source_bytes: &[u8], ++ doc_v1_json: &str, ++ mime: &str, ++ index: &I, ++) -> Result { ++ let (hash, verdict) = paperless_kv::preflight(source_bytes, index); ++ if let Verdict::Duplicate { matched } = verdict { ++ return Ok(Ingested::Duplicate { hash, matched }); ++ } ++ let ir = ogar_from_docv1::from_doc_v1(doc_v1_json, hash.0, mime).map_err(IntakeError::DocV1)?; ++ Ok(Ingested::Novel { ++ hash, ++ ir: Box::new(ir), ++ }) ++} ++ ++/// Pixel retina, full leg: a grey page → recognition → `doc.v1` → one `DocIr`, ++/// gated. Behind the off-by-default `ocr` feature. ++/// ++/// The gate runs before the executor is touched, so a duplicate page costs a ++/// hash and a lookup rather than a recognition pass. That ordering is the ++/// reason S-2 exists and it is asserted, not assumed ++/// (see `duplicate_never_reaches_the_retina`). ++/// ++/// # Errors ++/// [`IntakeError::Ocr`] if recognition fails, [`IntakeError::NotADocument`] if ++/// the executor answers with another response shape, [`IntakeError::DocV1`] if ++/// its own `doc.v1` does not pass the seam. ++#[cfg(feature = "ocr")] ++pub fn ingest_image( ++ grey: &[u8], ++ width: usize, ++ height: usize, ++ mime: &str, ++ executor: &tesseract_ogar::OcrExecutor, ++ index: &I, ++) -> Result { ++ use tesseract_ogar::{BinarizeMode, OcrRequest, OcrResponse}; ++ ++ let (hash, verdict) = paperless_kv::preflight(grey, index); ++ if let Verdict::Duplicate { matched } = verdict { ++ return Ok(Ingested::Duplicate { hash, matched }); ++ } ++ let resp = executor ++ .execute(OcrRequest::RecognizeDocument { ++ grey, ++ width, ++ height, ++ with_dict: false, ++ harvest_profile: None, ++ binarize: BinarizeMode::default(), ++ }) ++ .map_err(IntakeError::Ocr)?; ++ let OcrResponse::DocumentOut { doc_json, .. } = resp else { ++ return Err(IntakeError::NotADocument); ++ }; ++ let ir = ogar_from_docv1::from_doc_v1(&doc_json, hash.0, mime).map_err(IntakeError::DocV1)?; ++ Ok(Ingested::Novel { ++ hash, ++ ir: Box::new(ir), ++ }) ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ use paperless_kv::DocumentGuid; ++ use std::cell::Cell; ++ ++ /// An index that knows nothing. ++ struct Empty; ++ impl DedupIndex for Empty { ++ fn look_up(&self, _: &ContentSha256) -> Option<(DocumentGuid, MatchedOn)> { ++ None ++ } ++ } ++ ++ /// An index that has seen everything, and counts how often it was asked. ++ struct SeenAll { ++ asked: Cell, ++ matched: MatchedOn, ++ } ++ impl DedupIndex for SeenAll { ++ fn look_up(&self, _: &ContentSha256) -> Option<(DocumentGuid, MatchedOn)> { ++ self.asked.set(self.asked.get() + 1); ++ Some(( ++ DocumentGuid(lance_graph_contract::facet::FacetCascade::default()), ++ self.matched, ++ )) ++ } ++ } ++ ++ /// The DOM fixture. Gated with its tests: without `dom` nothing reads it, ++ /// and an unused-const warning in the DEFAULT build would be noise a real ++ /// warning could hide behind. ++ #[cfg(feature = "dom")] ++ const PAGE: &str = "
Acme GmbH
Invoice body text \ ++ here
"; ++ ++ #[cfg(feature = "dom")] ++ #[test] ++ fn dom_retina_produces_one_shape() { ++ let out = ingest_html(PAGE.as_bytes(), &Empty).expect("utf-8"); ++ let ir = out.ir().expect("novel"); ++ assert_eq!(ir.source, ogar_doc_ir::Provenance::Dom); ++ assert_eq!(ir.version, ogar_doc_ir::DOC_IR_VERSION); ++ // And it passes the IR's OWN load gate, which is the contract both ++ // retinas are held to. ++ let json = ogar_doc_ir::to_json(ir).expect("serialize"); ++ assert!(ogar_doc_ir::from_json(&json).is_ok()); ++ assert!( ++ !ir.pages[0].regions.is_empty(), ++ "the DOM landmarks must become regions, or this proves nothing" ++ ); ++ } ++ ++ #[cfg(feature = "dom")] ++ #[test] ++ fn dom_identity_matches_the_gate() { ++ // `spider_doc_ir::harvest` hashes `content.as_bytes()`; `preflight` ++ // hashes the same bytes. If these ever diverge, a document's gate ++ // identity and its IR identity would disagree — silently. ++ let out = ingest_html(PAGE.as_bytes(), &Empty).expect("utf-8"); ++ let ir = out.ir().expect("novel"); ++ assert_eq!( ++ &ir.content_sha256, ++ &out.hash().0, ++ "the retina's identity and the S-2 gate's must be the same hash" ++ ); ++ } ++ ++ #[cfg(feature = "dom")] ++ #[test] ++ fn duplicate_never_reaches_the_retina() { ++ // Deliberately NOT valid HTML and NOT valid UTF-8 tail: if the retina ++ // ran at all, `ingest_html` would have to decode these bytes and would ++ // fail. Reaching `Duplicate` proves the gate short-circuits FIRST, ++ // which is the whole of S-2. ++ let bytes = b"\xff\xfe not utf-8 at all"; ++ let index = SeenAll { ++ asked: Cell::new(0), ++ matched: MatchedOn::Original, ++ }; ++ let out = ingest_html(bytes, &index).expect("the gate answers before decoding"); ++ assert!(matches!(out, Ingested::Duplicate { .. })); ++ assert_eq!(index.asked.get(), 1, "the gate is asked exactly once"); ++ // The same bytes as a NOVEL input must fail — proving the test above ++ // is not passing because the bytes are somehow acceptable. ++ assert!(matches!( ++ ingest_html(bytes, &Empty), ++ Err(IntakeError::NotUtf8(_)) ++ )); ++ } ++ ++ #[cfg(feature = "dom")] ++ #[test] ++ fn derived_artifact_match_is_reported_as_such() { ++ let index = SeenAll { ++ asked: Cell::new(0), ++ matched: MatchedOn::Derived, ++ }; ++ let out = ingest_html(PAGE.as_bytes(), &index).expect("gate"); ++ assert!(matches!( ++ out, ++ Ingested::Duplicate { ++ matched: MatchedOn::Derived, ++ .. ++ } ++ )); ++ } ++ ++ #[test] ++ fn doc_v1_seam_fails_loud_on_a_drifted_producer() { ++ let good = r#"{"schema":"tesseract-rs/doc.v1","pages":[{"page":0,"width":100, ++ "height":100,"regions":[{"type":"text","bbox":[0,0,50,50],"lines":[]}]}]}"#; ++ let img = b"pretend-these-are-image-bytes"; ++ let ok = ingest_doc_v1(img, good, "image/png", &Empty).expect("valid doc.v1"); ++ let ir = ok.ir().expect("novel"); ++ assert_eq!(ir.source, ogar_doc_ir::Provenance::Ocr); ++ assert_eq!(ir.mime, "image/png"); ++ // Identity is over the IMAGE bytes, not the JSON: two renderings of one ++ // scan must not look like two documents. ++ assert_eq!(&ir.content_sha256, &ContentSha256::of(img).0); ++ ++ // An off-vocabulary region kind is refused at the seam. ++ let drifted = good.replace(r#""type":"text""#, r#""type":"paragraph""#); ++ assert!(matches!( ++ ingest_doc_v1(img, &drifted, "image/png", &Empty), ++ Err(IntakeError::DocV1(_)) ++ )); ++ // ...and so is a wrong schema marker. ++ let wrong = good.replace("tesseract-rs/doc.v1", "tesseract-rs/doc.v2"); ++ assert!(matches!( ++ ingest_doc_v1(img, &wrong, "image/png", &Empty), ++ Err(IntakeError::DocV1(_)) ++ )); ++ } ++ ++ #[test] ++ fn a_duplicate_never_reaches_the_doc_v1_seam() { ++ // The pixel leg's own S-2 proof. The JSON below is garbage: if the ++ // seam ran at all, `from_doc_v1` would refuse it and this would be ++ // `Err(DocV1)`. Landing on `Duplicate` proves the gate answered ++ // FIRST — which is the whole of S-2, and is a per-entry-point ++ // property, not something the DOM leg's test can establish for this ++ // one. ++ let junk = "}{ not json"; ++ let img = b"already-seen-image-bytes"; ++ let index = SeenAll { ++ asked: Cell::new(0), ++ matched: MatchedOn::Original, ++ }; ++ let out = ingest_doc_v1(img, junk, "image/png", &index).expect("the gate answers first"); ++ assert!(matches!(out, Ingested::Duplicate { .. })); ++ assert_eq!(index.asked.get(), 1, "the gate is asked exactly once"); ++ // ...and the same junk as a NOVEL input must fail, or the assertion ++ // above would pass for the wrong reason. ++ assert!(matches!( ++ ingest_doc_v1(img, junk, "image/png", &Empty), ++ Err(IntakeError::DocV1(_)) ++ )); ++ } ++ ++ #[cfg(feature = "dom")] ++ #[test] ++ fn both_retinas_land_in_one_type() { ++ // The point of the whole crate: a web page and a scan are the same ++ // Rust type, distinguishable only by the field that says so. ++ let dom = ingest_html(PAGE.as_bytes(), &Empty).expect("utf-8"); ++ let pixel = ingest_doc_v1( ++ b"img", ++ r#"{"schema":"tesseract-rs/doc.v1","pages":[{"page":0,"width":10,"height":10, ++ "regions":[{"type":"text","bbox":[0,0,5,5],"lines":[]}]}]}"#, ++ "image/png", ++ &Empty, ++ ) ++ .expect("valid doc.v1"); ++ let irs: Vec<&DocIr> = vec![dom.ir().expect("novel"), pixel.ir().expect("novel")]; ++ assert_eq!(irs.len(), 2); ++ assert_eq!(irs[0].source, ogar_doc_ir::Provenance::Dom); ++ assert_eq!(irs[1].source, ogar_doc_ir::Provenance::Ocr); ++ assert!(irs.iter().all(|i| i.version == ogar_doc_ir::DOC_IR_VERSION)); ++ } ++} +-- +2.43.0 + diff --git a/.claude/harvest/paperless-token-plateau/README.md b/.claude/harvest/paperless-token-plateau/README.md new file mode 100644 index 000000000..e03a3097c --- /dev/null +++ b/.claude/harvest/paperless-token-plateau/README.md @@ -0,0 +1,115 @@ +# paperless-token plateau — an ARCHIVE, no longer the way to get this code + +> **⊘ SUPERSEDED 2026-08-24 — read this first.** This code now lives in +> `AdaWorldAPI/tesseract-rs` as `crates/tesseract-paperless`, feature-gated +> (`default` = the S-2 gate + `doc.v1` → `DocIr`; `ocr` = the in-process +> recognizer; `token` = the lexical seam). tesseract-rs accepts pushes and +> runs CI on all three tiers, so **that is the one source of truth.** Nothing +> should be reconstructed from this patch except as forensics — and nothing +> should be synced back to paperless-rs, which is a dead copy. +> +> Two corrections carried by the move, both worth keeping: the `[patch]` +> section is gone entirely (tesseract-rs path-deps its siblings, so the +> escaping-relative-path trap does not arise), and `ingest_html` was replaced +> by a producer-agnostic `ingest_doc_ir(bytes, index, build)` — the caller +> brings its own producer as a closure, so the broken `spider_doc_ir` build is +> no longer anyone's dependency. +> +> **A correction to this file's own table:** it stated +> `coca_academic_20k.tsv` as 180 361 bytes. Measured, it is **226 651**. The +> sha256 column was right, which is why the reconstruction check passed — but +> the verification I ran covered *patch-applies* and *tests-pass*, not the byte +> counts, and saying "verified" without that scope was too broad. + +`PROBE-TOKEN-SEAM-1` and its architecture doc live in +`AdaWorldAPI/paperless-rs`. That repo is in this session's scope for READS but +its push is denied at the org/GitHub-App level: + +``` +remote: Claude doesn't have GitHub access to AdaWorldAPI/paperless-rs for your organization. +fatal: unable to access 'https://github.com/AdaWorldAPI/paperless-rs/': ... 403 +``` + +Verified twice — through the session proxy AND with the proxy bypassed. The +in-environment `GH_TOKEN` is a 14-character placeholder, not a credential, so +the documented "a 403 here is usually the proxy" escape does not apply: this one +is real. The commit therefore exists only in an ephemeral container, and +`E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1` cites evidence that would die with it. +This directory is that insurance, following the plateau pattern tesseract-rs +already documents for a denied repo. + +## What is here + +`0001-token-seam.patch` — **all four commits, minus the two corpus fixtures**, +which are 350 KB of bytes exactly reproducible by two commands (below). +Everything else is in the patch verbatim: +`crates/paperless-token/{src,examples,README.md,Cargo.toml}`, +`crates/paperless-intake/{src,Cargo.toml}`, +`docs/TOKEN-SEAM-ARCHITECTURE.md`, the root `Cargo.toml` wiring, and the +`CLAUDE.md` update. + +| # | commit | what it adds | +|---|---|---| +| 1 | `one tokenization receipt, three borrowed consumers` | the seam probe: contract, lane, view, the three consumer surfaces, 41 gates | +| 2 | `re-cut the seam onto ogar-doc-ir` | the receipt mints no identity — `SpanKey` is read from the document layer's own IR | +| 3 | `two measurement probes` | carrier width (flat u8 vs the 256:256 rail) and WordId vs TokenId in the tile | +| 4 | `wire both retinas into intake` | `paperless-intake` stops being a stub: the S-2 gate in front of the DOM and pixel producers | + +## Reconstruction + +```sh +git -C paperless-rs checkout -b claude/bpe-tokenization-architecture-3xd4eh +git -C paperless-rs am 0001-token-seam.patch # four commits + +# the two fixtures the patch deliberately omits: +mkdir -p paperless-rs/crates/paperless-token/corpus +cp tantivy/benches/alice.txt paperless-rs/crates/paperless-token/corpus/alice.txt +awk -F, 'NR>1 {print $4"\t"$5}' \ + lance-graph/crates/deepnsm/word_frequency/academic_20k.csv \ + > paperless-rs/crates/paperless-token/corpus/coca_academic_20k.tsv +``` + +Then verify, because a reconstructed fixture that differs by one byte changes +every measured number in the report: + +| file | bytes | sha256 | +|---|---|---| +| `corpus/alice.txt` | 174 693 | `15124d40c182677c2d90fba80310173d63428e0591ce0df3e9bdc01a789a89c6` | +| `corpus/coca_academic_20k.tsv` | 226 651 | `4ae20ce39dd3018346700e0f88df2b59e1a7df4e4a06e0f285fd44065166e0f0` | +| source `academic_20k.csv` | — | `1dfd5edaa5a6ac9b8ac5abbf87894abaf7de8a449ad7c09ca1f6324226396e2d` | + +`alice.txt` is Project Gutenberg's *Alice's Adventures in Wonderland* (public +domain), carried from `tantivy/benches/alice.txt`. It is CRLF with a BOM — the +probe normalises both, and the gate that would otherwise have hidden a failure +to do so is `T-CORPUS`. + +## Running it + +```sh +cargo run --release -p paperless-token --example probe_token_seam +``` + +Expected: `PROBE-TOKEN-SEAM-1: ALL 41 GATES GREEN`, and the four measured +tables the epiphany quotes. + +Note the second commit adds an `ogar-doc-ir` dependency, so the reconstruction +needs the workspace `Cargo.toml` from the patch (it is in there) and network +access to OGAR. That pair floats on `branch = "main"` rather than a rev, and +the patch's own manifest comment says why: cargo does not unify a `branch` and +a `rev` source even at the same commit, and two SourceIds for `ogar-doc-ir` +means two incompatible `DocIr` types in one binary. + +`paperless-intake` compiles with `--features ocr` (the in-process recognizer) +but NOT with `--features dom`: `spider_doc_ir` omits `TableCell::confidence`, +a field `ogar-doc-ir` added after spider's only commit. That is an upstream +break, recorded in full in `crates/paperless-intake/Cargo.toml`, not something +the reconstruction did wrong. + +## This patch was verified, not assumed + +A banked patch that does not apply is not insurance. Checked before landing: +`git am` of all four onto the base commit applies clean, and +`cargo test -p paperless-intake -p paperless-kv` on the reconstructed tree is +green (2 + 6 tests). The `paperless-token` probe is NOT part of that check — +it needs the two corpus fixtures the patch deliberately omits, so run it only +after the reconstruction step above.