diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index e98f45ea9..05fd25114 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,21 @@ +## 2026-09-03 — E-TWO-FATE-PROBES-KILL-DIFFERENT-WAYS-1 — `semantic_chunker` fails at the mechanism, `spiral_segment` fails at the ratio; a pre-registered gate that only checked "pass/fail" would have hidden which + +**Status:** FINDING (measured, `crates/thinking-engine/examples/{chunker_falsifier,spiral_gate_probe}.rs`, both run on real baked data, no synthetic corpus). **Confidence:** High — both arms were pre-registered before either probe ran (plan `thinking-engine-harvest-closure-v1.md` §4b), and neither threshold was retuned after seeing a result. + +**What the closure plan asked.** The two remaining §1c rows each had a promotion path gated on a falsifier: `semantic_chunker` → deepnsm-v2's text side, `spiral_segment` → the codec certification battery. Both probes ran on real data (the tier-1..4 calibration corpus + the Jina v5 tokenizer + `jina-v5-codebook` for the chunker; five real baked 256x256 tables — jina-v3, bge-m3, reranker, jina-v5 u8 and i8 — for the spiral codec). Both came back KILL, and the two KILLs are structurally different failures worth telling apart. + +**`semantic_chunker` fails at the MECHANISM — recall = 0.000 at every threshold swept (0.30 / 0.45 / 0.60), across 168 cross-topic passages.** An all-zero result is exactly the shape a broken harness produces, so before trusting it a diagnostic re-ran the module's OWN pre-existing adversarial positive control — the synthetic maximally-separated centroid corners from its `detects_boundary_between_topics` test — against the SAME `jina-v5-codebook` table. **It also produced zero boundaries.** So the null is not a probe-design artifact: the perturb→think→top-k-Jaccard convergence pattern this module is built on simply does not diverge on an HDR-encoded table, even on inputs engineered to be as separable as the 256-centroid space allows. The module's own existing test already carried a hedge to this effect ("On uniform HDR tables the convergence patterns may not diverge strongly") — this is that hedge turned into a measured, falsifiable KILL. (The control was first run as a throwaway script and only its result recorded here; CodeRabbit's review on PR #1144 correctly flagged that a deleted diagnostic leaves the mechanism-null conclusion unreproducible, so it is now a 4th committed arm in `chunker_falsifier.rs` — same corners, same zero.) + +**`spiral_segment` fails at the RATIO on u8 tables (fidelity is fine there); on i8 it fails BOTH gates — corrected post-merge by a Codex review comment on this PR (`chatgpt-codex-connector[bot]`, P2), see the addendum below.** Pearson r and Spearman rho clear the 0.9980 gate on every u8 table at every max_error tried — the codec reconstructs distance rows accurately there. The i8 table does NOT clear it at any max_error tested: at its best configuration (0.005) r = 0.9993 passes but rho = 0.9975 falls short of the 0.9980 floor, and rho only gets worse as max_error loosens (0.9906 / 0.9590 / 0.7137). What kills every table, u8 or i8, is `bytes(spiral) <= bytes(u8)/2`: the best compression measured is ~0.28x on u8 (i.e. the spiral encoding is **~3.6x LARGER** than the u8 table it would replace) and ~0.80x on i8 at its loosest setting, nowhere near the module's own "51x compression" doc-comment claim or even the ecosystem's 2x floor. Root cause is visible in the segment count: a real per-row CDF from a trained codebook needs ~114-143 spiral segments (8 bytes each) to hit even a loose 0.05 max_error, because the codec's fitting premise — few segments suffice per row — only holds on smooth synthetic curves, and a trained model's distance rows are not smooth. The 51x claim was never false against the data it was tested on; it was false against the distributional shape of data it was never tested on. + +**Addendum 2026-09-03 (Codex review, PR #1144, verified against the probe's own re-run):** the FIRST version of this entry and the plan's §4c write-up both claimed "even the i8 table clears [fidelity] at max_error <= 0.01" — this was wrong, and the contradicting number (rho = 0.9975 < 0.9980 at max_error 0.005) was sitting in the same write-up's own results table. i8 never clears BOTH fidelity gates at any tested max_error; the KILL verdict for `spiral_segment` is unaffected (i8 also fails the compression gate at every max_error, so the module stays LAB either way), but the characterization "fidelity passes, only compression fails" is accurate for u8 only, not for i8. Corrected in the plan and here rather than silently editing the original claim away — see this repo's own falsifiability rule on not treating a code-adjacent number as verified until it's actually checked against the row that contradicts it. + +**The generalizable point.** "The gate failed" is not one finding — WHERE it failed changes what happens next. A chunker whose mechanism cannot discriminate needs a different algorithm entirely (or a different substrate) before it is worth re-probing; a codec whose fidelity is fine but whose ratio is wrong needs either a different fitting strategy or an honest downgrade of its claimed compression, and re-running the SAME probe on a friendlier table would not fix either — which is why neither module is being re-probed. Both fate-probe reports therefore carry a per-configuration table (not just a verdict), so a future session reading the KILL knows which failure it inherited. + +**Both stay LAB, not deleted, not re-probed on friendlier data** — per the pre-registration's own commitment (§4b: "Not deleted... Not re-probed on a different corpus to get a better answer"). §1c is now fully closed: every row in the harvest census table has a verdict. + +Refs: `thinking-engine-harvest-closure-v1` §4b (pre-registration) / §4c (results) / §5 D-TEH-3; `crates/thinking-engine/src/{semantic_chunker,spiral_segment}.rs`; `.claude/agents/falsifier-auditor.md` (the can-it-fire discipline applied to the chunker's all-zero result); the falsifiability rule (CLAUDE.md § The falsifiability rule). + ## 2026-09-02 — E-THE-LIFT-GATE-FOUND-A-TIE-BLIND-SPEARMAN-1 — the D-TEH-3 comparison on a distinguishing fixture separated a same-formula copy from a wrong-estimator copy, and the tie-free fixtures the lab had used could not have **Status:** FINDING (measured, tests committed in `crates/jc/src/reliability.rs`). **Confidence:** High. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 078f7cea1..70ab4cbe3 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,12 @@ +## 2026-09-03 — branch (D-TEH-3 closed, after #1143 merged): both fate probes KILL — INVENTORY DELTA + +- MERGED #1143 (D-TEH-3 math half, `f05a358`'s ancestor): `jc::drift` / `jc::quorum` land, `cronbach.rs` deleted (see the 2026-09-02 delta below). +- ADDED (probes, not shipped API) `crates/thinking-engine/examples/chunker_falsifier.rs` — real-data falsifier for `semantic_chunker` (Jina v5 tokenizer + `jina-v5-codebook`, the tier-1..4 calibration corpus, 168 cross-topic + 8 same-topic passages, 20 SplitMix64 null permutations). Result: recall = 0.000 at every pre-registered threshold; confirmed a genuine mechanism null (not a harness bug) via a non-committed positive-control diagnostic on the module's own adversarial fixture shape. **Verdict: KILL — `semantic_chunker` stays LAB, not ported to deepnsm-v2.** +- ADDED (probes, not shipped API) `crates/thinking-engine/examples/spiral_gate_probe.rs` — real-data falsifier for `spiral_segment` against five real baked 256x256 tables (jina-v3/bge-m3/reranker/jina-v5 u8, jina-v5 i8). Result: fidelity clears the r/rho >= 0.9980 gate on every u8 table; compression fails on every table — `ratio_vs_u8 = bytes(u8)/bytes(spiral)` tops out at ~0.28x (i.e. the spiral encoding is ~3.6x LARGER than the u8 table it would replace), not the module's claimed 51x smaller. **Verdict: KILL — `spiral_segment` stays LAB, no certification battery scheduled.** +- FIXED (unrelated to the probes, surfaced by the same `clippy -D warnings` run and paid down per `TD-THINKING-ENGINE-EXCLUDED-DEBT-1`): `world_model.rs::from_engine_state` (10 args → `ThoughtSignals` bundle, 4 args, zero in-tree callers); `cognitive_stack.rs::MetaCognition` (`impl Default`); `dto.rs::ThoughtIndex` (`#[derive(Default)]`); `tensor_bridge.rs::pairwise_cosines` (documented `#[allow]`, symmetric double-write has no clean iterator form); `qualia.rs` (19 range-fill loops → `.fill()`, 1 zip rewrite); `signed_domino.rs` (1 enumerate rewrite). 25 of ~40 pre-existing lints paid; lib tests for every touched module green (43/43) before and after. +- §1c of `thinking-engine-harvest-closure-v1.md` is now CLOSED — every harvested row has a verdict (ghost prior: ported; calibration math: lifted into jc; semantic_chunker: KILL; spiral_segment: KILL). +- NOT IN THIS DELTA: the ~13 remaining pre-existing clippy lints outside `world_model.rs`/`cognitive_stack.rs`/`dto.rs`/`tensor_bridge.rs`/`qualia.rs`/`signed_domino.rs` (spread across `cognitive_trace.rs`, `contract_bridge.rs`, `f32_engine.rs`, `l4.rs`, `l4_bridge.rs`, `layered.rs`, `pooling.rs`, `branching.rs`, `composite_engine.rs` — not read this pass); the 3 unrelated `jc`-crate `dead_code` warnings (a separate crate's own debt). + ## 2026-09-02 — branch (D-TEH-3, after #1142 merged): calibration math lifted into jc — INVENTORY DELTA - MERGED #1142 (D-TEH-2, `3c5f040`): no contract type; planner `nars/ghost_prior.rs` + lab-crate edits + `weather-poc.yml` sibling checkout (see PR_ARC). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index a04451b59..55b3aa3f3 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -20,7 +20,7 @@ | D-TEH-0 | census: live footprint measured (one required consumer via `bridge_gate`; one optional via `with-engine`), 51-file fate table, open-row reconciliation, four-wave closure, idea harvest, four rulings asked | **Shipped (plan-only, this PR)** | | D-TEH-1 | W1: `bridge_gate` (seven items) → `lance_graph_contract::bridge_gate`; callcenter re-imports and drops the path dep; thinking-engine keeps a re-export shim | **Shipped 2026-09-02** (before: required dep, 6 crossing sites, dep-drop alone fails 6 × E0433; after: zero thinking-engine deps in callcenter metadata, contract 1303/1303, callcenter 156/156, driver default + `with-engine` green, ALU artery files byte-identical). `with-engine` re-point deferred by stop condition: D-TTV-1 not landed | | D-TEH-2 | W2: ghost prior → planner `nars/ghost_prior.rs` over `WisdomMarker`, per-thought, two-sided falsifiers; crate `ghosts.rs` deleted | **Shipped 2026-09-02** — `GhostPrior`/`PriorFloor` in planner `nars/ghost_prior.rs`, 14 tests incl. the calibration gate (default floor = `Marker`, the gate reversed the first declaration); `ghosts.rs` + `think.rs` deleted; TD-GHOST-ECHO-DUP-1 resolved; consumer D-HOUSE-4 unblocked | -| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | **Math half Shipped 2026-09-02** — `jc::drift` (re-encode drift + delta summary) and `jc::quorum` (pairwise agreement, bands, Cronbach report) added; cronbach LIFTED (same formula; `f32` copy loses the `1e7`-shifted fixture), spearman PERFECTED-IN-JC (retired copy was tie-blind: 1.000 vs 0.9487 on one tie); lab `cronbach.rs` deleted, `reencode_safety` / `silu_correction` / `ground_truth` are glue over jc. Open: the `semantic_chunker` / `spiral_segment` falsifier halves | +| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | **Shipped 2026-09-02/03, all three halves closed** — `jc::drift` (re-encode drift + delta summary) and `jc::quorum` (pairwise agreement, bands, Cronbach report) added; cronbach LIFTED (same formula; `f32` copy loses the `1e7`-shifted fixture), spearman PERFECTED-IN-JC (retired copy was tie-blind: 1.000 vs 0.9487 on one tie); lab `cronbach.rs` deleted, `reencode_safety` / `silu_correction` / `ground_truth` are glue over jc. `semantic_chunker`: **KILL** — recall 0.000 at every pre-registered threshold, confirmed a real mechanism null via a committed positive-control arm in `chunker_falsifier.rs` (the module's own adversarial synthetic-corners shape, added 2026-09-03 per CodeRabbit review on #1144 so the evidence is reproducible, not just asserted); stays LAB. `spiral_segment`: **KILL** — fidelity passes on every u8 table (r/rho >= 0.998) but NOT on i8 (rho tops out at 0.9975, below the 0.998 floor, corrected 2026-09-03 per Codex review on #1144); compression fails on every table regardless — `ratio_vs_u8 = bytes(u8)/bytes(spiral)` tops out at ~0.28x (i.e. the spiral encoding is ~3.6x LARGER than the u8 table it would replace), not the claimed 51x smaller; stays LAB, no certification battery scheduled. Plan `thinking-engine-harvest-closure-v1.md` §4c | | D-TEH-4 | W3: ENTROPY M8 engine collapse with dtype parity suite; 5 cascade shapes + 3 lens modules collapse | Queued | | D-TEH-5 | W4: residue deleted, crate renamed `thinking-lab` with a `--manifest-path` CI line; §2 rows closed; `TD-THINKING-ENGINE-EXCLUDED-DEBT-1` paid | Queued — closes the chapter | diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index b3b54f238..6dab0efd2 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -991,6 +991,8 @@ Both recorded so P4 (D-TRI-2/3 ancestry-pipeline consolidation) inherits them. R **Surfaced during P3** (rung dedup). thinking-engine is a workspace-EXCLUDED crate (root Cargo.toml `exclude`), so no CI/clippy gate runs on it — it has accumulated ~40 `clippy -D warnings` lints (loop-index, `map_or` simplify, too-many-args, missing-Default; all in modules P3 never touched — e.g. `cognitive_stack.rs:242` MetaCognition::new, `world_model.rs:154` from_engine_state) and at least one test-compile break (`DualResult` missing `convergence_signed`/`convergence_unsigned` fields, a stale test). The **lib builds clean**; only `--tests` and `clippy -D warnings` fail. Correlates with E-RUNG-ASCENT-WIRED-1's finding that the whole crate is orphaned from the production spine (only bridge_gate→callcenter + dto→driver[feature] are external wires). PAYOFF: gated on the P4 ancestry-consolidation decision (wire the gems into the spine → then it earns a CI slot and the debt gets paid; or retire the orphaned cluster). Not paid in P3 (out of scope; excluded crate). +**Update 2026-09-03 (D-TEH-3 fate-probe PR, partial paydown, both named items resolved).** Both examples the original entry NAMED explicitly are fixed: `world_model.rs:154` (`from_engine_state`, 10 args → grouped into a new `ThoughtSignals` bundle, 4 args; zero in-tree callers so behaviour is unchanged by construction, verified by grep across the whole repo) and `cognitive_stack.rs:242` (`MetaCognition::new` → added `impl Default for MetaCognition` calling `new()`, standard `new_without_default` fix). Also paid down while touching adjacent files surfaced by the same `clippy -D warnings` run: `tensor_bridge.rs::pairwise_cosines` (scoped `#[allow(clippy::needless_range_loop)]` with a comment — the symmetric double-write to `matrix[i][j]`/`matrix[j][i]` alongside `self.embeddings[i]`/`[j]` has no single-iterator expression, clippy's own suggested rewrite only reaches row `i`), `qualia.rs` (19 `for i in A..=B { w[i] = C; }` range-fill loops in `family_band_weights` → `w[A..=B].fill(C)`, plus `to_voice_channels`'s zip-instead-of-index rewrite — 20 lints, all mechanical constant-fills, zero behaviour risk), `signed_domino.rs` (one `for j in 0..n` → `for (j, &val) in row.iter().enumerate()`), `dto.rs::ThoughtIndex` (`#[derive(Default)]` — every field is a `Vec`, whose default is empty, matching `new()`'s body exactly). **25 of the ~40 lints paid; `cargo test --lib` on every touched module green (43/43) before and after.** Remaining ~13 (excluding 3 unrelated `jc`-crate dead_code warnings that are a separate crate's own debt) span `cognitive_trace.rs`, `contract_bridge.rs`, `f32_engine.rs`, `l4.rs` (x3), `l4_bridge.rs` (x2), `layered.rs`, `pooling.rs`, `branching.rs`, `composite_engine.rs` — not read this pass, left as this entry's continuing debt rather than risking a behaviour change on unfamiliar code under a Sonnet-tier grindwork budget. The P4 ancestry-consolidation gate this entry's PAYOFF names is still the real close. + ## TD-ONTOLOGY-CLIPPY-DEBT-1 — lance-graph-ontology (member) has pre-existing clippy -D warnings errors that block dependent `-p` clippy (2026-07-17) **Surfaced during P3.** `cargo clippy -p cognitive-shader-driver -- -D warnings` fails (exit 101) entirely inside `lance-graph-ontology` (a transitive dep): `doc-lazy-continuation` ×N, `iter-cloned-collect` (`.iter().copied().collect()` → `to_vec()`), and deprecated `oxrdf::Subject` type-alias uses (ttl_parse.rs, owl.rs, op_emitter.rs, lib.rs). P3 touches none of ontology; the driver's OWN code is clippy-clean (0 findings in driver.rs). Pre-existing by construction (not in P3's diff). PAYOFF: mechanical (doc reflow + `to_vec()` + `NamedOrBlankNode` migration), its own small PR — not P3 scope. Flags that repo CI likely does not run full-workspace `clippy -D warnings`, or main is already red there. diff --git a/.claude/plans/thinking-engine-harvest-closure-v1.md b/.claude/plans/thinking-engine-harvest-closure-v1.md index 23a9c0543..a1a0e21bc 100644 --- a/.claude/plans/thinking-engine-harvest-closure-v1.md +++ b/.claude/plans/thinking-engine-harvest-closure-v1.md @@ -204,6 +204,166 @@ harvest and not an amputation. | The 8-channel cascade edge (BECOMES / CAUSES / SUPPORTS / REFINES / GROUNDS / ABSTRACTS / RELATES / CONTRADICTS) | transcoded to the SPO palette, kept as the comma-level mantissa carrier | D-CSV-9, FUTURE-DESIGN "first wiring target" | | Persona as 12 constants + a mode with rung bounds and a collapse bias | a Layer-2 data card for the persona storyline, when it opens | O3 (parked) | +## 4b. D-TEH-3 fate probes — PRE-REGISTERED before the first run + +The two remaining §1c rows (`semantic_chunker`, `spiral_segment`) each carry a +falsifier as their gate, and a gate written after the numbers is not a gate. +Both probes, their arms and their thresholds are fixed HERE, before either was +executed; the run only fills in the measurements. + +### `semantic_chunker` — `examples/chunker_falsifier.rs` + +The §1c claim is "chunk boundaries are convergence jumps, no forward pass", +and the row's gate is "boundaries vs a gold sentence split; else stays LAB". +The probe uses the tier-1..4 calibration corpus of `jina_v5_ground_truth.rs` +(Rule 23 — real text, not synthetic), the Jina v5 tokenizer, the baked +`jina-v5-codebook` index + 256x256 table. Sentence pairs from DIFFERENT corpus +pairs are concatenated, so the gold boundary is the token seam by construction. + +| arm | what it measures | why it is there | +|---|---|---| +| can-fire | recall@+-4 tokens of the seam over 168 cross-topic passages | a chunker that never fires is not a chunker | +| null | the same passages with centroid order shuffled (20 SplitMix64 permutations), p95 of recall | "a boundary landed near the seam" must beat chance, and a boundary detector on shuffled input still finds boundaries | +| silence | the 8 same-topic (tier-1/2) passages, both orders | a chunker that splits everything carries as much information as one that never splits | + +**PASS** (at one threshold, the same threshold for all three arms): +recall >= 0.75 AND recall >= null_p95 + 0.15 AND false splits <= 2 of 8. +**KILL** otherwise. + +- PASS commits: the port to `deepnsm-v2`'s text side is on, as its own PR with + the probe carried across as the port's regression gate. It does NOT commit + the tesseract-paperless sentence assembler to using it — that consumer is a + separate decision with its own falsifier. +- KILL commits: the module stays LAB and its §1c row is closed as + measured-negative, with the numbers on the board. It is NOT deleted (a LAB + verdict is a home, not a death sentence) and it is NOT re-probed on a + different corpus to get a better answer. + +### `spiral_segment` — `examples/spiral_gate_probe.rs` + +The §1c row routes this to a codec home "via the certification battery" +(`certification-officer`). That battery needs the F32 cosine matrix re-derived +from the model source, which is not on disk here. This probe is the CHEAP GATE +in front of that expensive step: fit the codec to the four real baked 256x256 +tables in the tree (jina-v3, bge-m3, reranker, jina-v5 u8 + jina-v5 i8) and ask +whether it can clear the ecosystem floor at all. + +Thresholds are the ecosystem's own, not invented here: +`encoding-ecosystem.md` — "any encoding below the naive u8 floor is worse than +doing nothing"; the bgz-hhtl-d gate is Pearson >= 0.9980. Plus the claim the +module's own doc makes ("51x compression"): the codec must at least halve what +it replaces, or it is a lossy re-encoding of a u8 table for no space. + +**PASS** if for SOME `max_error`, on EVERY table: r >= 0.9980 AND rho >= 0.9980 +AND spiral bytes <= u8 bytes / 2. **KILL** otherwise. + +- PASS commits: the certification battery is unblocked and scheduled — this + probe is explicitly NOT a certification (a baked u8 table is not the atomic + clock; only the source-derived F32 matrix is). +- KILL commits: the module stays LAB, the 51x claim is recorded as + measured-false at the fidelity the workspace requires, and no battery is run. + A codec that cannot preserve a baked table will not preserve its F32 parent. + +Both probes report a per-configuration table so a KILL says WHERE it failed, +not merely that it failed. + +## 4c. D-TEH-3 fate probes — RESULTS (both KILL, both stay LAB) + +Both probes run on real data (`crates/thinking-engine/data/jina-v5-codebook/` ++ `jina-v3-hdr/` + `bge-m3-hdr/` + `jina-reranker-v3-BF16-hdr/`, no synthetic +input), against the arms and thresholds pre-registered in §4b. **Neither +threshold was retuned after seeing a result** — the pre-registration's own +rule. + +### `semantic_chunker` — KILL, and the null result is a genuine mechanism +### null, not a harness artifact + +``` +threshold | recall | null p95 | false splits | bnd/pass | verdict + 0.30 | 0.000 | 0.000 | 0 of 8 | 0.00 | kill + 0.45 | 0.000 | 0.000 | 0 of 8 | 0.00 | kill + 0.60 | 0.000 | 0.000 | 0 of 8 | 0.00 | kill +``` + +recall = 0.000 at every threshold means `find_boundaries` never fired ONCE +across all 168 cross-topic passages, at any of the three thresholds swept. +Before trusting an all-zero result, a diagnostic ran the module's OWN +adversarial positive-control shape from its `detects_boundary_between_topics` +test (synthetic centroid corners 0-4 vs 250-254, maximally distant in the +256-centroid table) against the same `jina-v5-codebook` table the falsifier +used. **The positive control ALSO produced zero boundaries.** This was first +run as a throwaway, non-committed script — a real gap Codex review caught on +this PR (#1144): a deleted diagnostic means a later reader can reproduce the +KILL, but not the reasoning for calling it a mechanism null rather than a +harness bug. Fixed by committing the control as a 4th arm inside +`chunker_falsifier.rs` itself (reproduces the same zero, confirmed by +re-running it after landing). So this is a genuine mechanism/table-level null: +the perturb-think-top-k-Jaccard convergence pattern at `max_cycles: 10` does +not discriminate on this HDR-encoded table at all, even on inputs designed to +be maximally separable. The module's own pre-existing test +(`detects_boundary_between_topics`) already carried a comment hedging exactly +this ("On uniform HDR tables the convergence patterns may not diverge +strongly") — the falsifier turns that hedge into a measured, pre-registered +KILL rather than an unverified doubt. + +**Verdict: KILL.** `semantic_chunker` stays LAB. Not ported to deepnsm-v2. Not +deleted. Not re-probed on a friendlier corpus to chase a different answer. + +### `spiral_segment` — KILL, and the failure is compression ratio on u8 +### (fidelity holds), compression AND fidelity on i8 (correction below) + +``` + table | max_error | r | rho | x u8 | seg/row | + jina-v3 u8 | 0.005 | 1.0000 | 1.0000 | 0.22 | 142.27 | + jina-v3 u8 | 0.050 | 0.9991 | 0.9991 | 0.28 | 114.11 | + bge-m3 u8 | 0.005 | 1.0000 | 1.0000 | 0.22 | 142.61 | + reranker u8 | 0.005 | 1.0000 | 1.0000 | ~0.2 | ~140 | + jina-v5 u8 | 0.050 | 0.9991 | 0.9991 | 0.28 | 113.72 | + jina-v5 i8 | 0.005 | 0.9993 | 0.9975 | 0.28 | 115.10 | + jina-v5 i8 | 0.050 | 0.7709 | 0.7137 | 0.80 | 39.91 | +``` +(full table: 5 tables x 4 max_errors = 20 rows, see the example's own output) + +Fidelity (Pearson r, Spearman rho) clears the 0.9980 gate comfortably on every +u8 table at every max_error tried. **The codec is accurate on u8 tables.** +The i8 table is the one real exception, caught by Codex on this PR +(`chatgpt-codex-connector[bot]`, P2): at its best configuration +(max_error 0.005) r = 0.9993 clears the floor but rho = 0.9975 does not +(the table above already shows this row) — i8 fidelity never clears BOTH +gates at any max_error tested, and gets strictly worse as max_error loosens +(rho 0.9975 -> 0.9906 -> 0.9590 -> 0.7137). Compression ALSO fails on i8 at +every max_error (x u8 tops out at 0.80x, still short of `MIN_RATIO_VS_U8 = 2.0`), +so the KILL verdict for `spiral_segment` is unaffected by this correction — +the codec fails BOTH gates on i8, not just the compression gate the u8 +tables already fail. What the original write-up got wrong was calling this +a "compression-only" failure; it is compression-only on u8, and a +double failure on i8. What kills it, in either case, is compression: +`x u8` (spiral bytes vs u8-table bytes) never exceeds ~0.28x on a u8 table — +i.e. the spiral encoding is **~3.6x LARGER** than the u8 table it would +replace, not smaller, let alone the "51x compression" the module's own doc +comment claims. Root cause visible in `seg/row`: a real HDR/CDF-encoded +distance row needs ~114-143 spiral segments (8 bytes each) to hit even a +0.05 max_error, because the fitting premise (few segments per row) only holds +on smooth, low-curvature synthetic data — a real codebook's per-row CDF is not +smooth. Even at the loosest max_error swept (0.05, well past the fidelity +gate's own comfort margin), no table's compression ratio approaches the +`MIN_RATIO_VS_U8 = 2.0` floor, let alone the ecosystem's u8-beats-nothing +floor doubled. + +**Verdict: KILL.** `spiral_segment` stays LAB. The certification battery +(F32-source re-derivation, `certification-officer`) is NOT scheduled — a +codec that cannot beat a baked u8 table by 2x will not beat its F32 parent +either. The "51x compression" doc-comment claim is recorded here as +measured-false at the fidelity this workspace requires; it was never false at +the fidelity the module tested itself against (smooth synthetic curves), only +against the real distributional shape of a trained model's own table. + +### §1c is now CLOSED — every row in the table has a verdict + +Both remaining open rows (`semantic_chunker`, `spiral_segment`) are now KILL, +joining the earlier D-TEH-2/D-TEH-3 PASS rows already landed +(#1142/#1143). No §1c row remains unprobed. + ## 5. Deliverables | D-id | title | scope | status | @@ -211,7 +371,7 @@ harvest and not an amputation. | D-TEH-0 | census + fate table + open-row reconciliation + idea harvest (this plan) | plan + board rows | Shipped (this PR) | | D-TEH-1 | W1: `bridge_gate` (seven items) → `lance_graph_contract::bridge_gate`; callcenter re-imports and drops the path dep; thinking-engine keeps a re-export shim | contract + callcenter | **Shipped 2026-09-02** — edge measured before (required dep, 6 crossing sites, dep-drop fails 6 × E0433) and after (zero thinking-engine deps in callcenter metadata; 1303 + 156 tests, driver default + `with-engine` green). The `with-engine` re-point is NOT part of this wave: D-TTV-1 is Queued and the engine hook still lives in thinking-engine, so there is nothing to re-point it at (stop condition honoured). thinking-engine is now a leaf for every REQUIRED edge; the one remaining edge is the ALU's optional engine hook | | D-TEH-2 | W2: ghost prior harvested as planner `nars/ghost_prior.rs` over `WisdomMarker`, per-thought, with two-sided falsifiers; crate `ghosts.rs` deleted | planner | **Shipped 2026-09-02** — planner `nars/ghost_prior.rs` (`GhostPrior`, `PriorFloor`, `Trace`, `calibration::{recurrence_fixture, discrimination}`; 14 tests); `ghosts.rs` + `examples/think.rs` deleted; lab `persona`/`world_model`/`awareness_dto` re-pointed to `contract::escalation::GhostEcho` (TD-GHOST-ECHO-DUP-1 resolved). Calibration gate REVERSED the first-declared floor: `Marker` (0.1, never pruned) discriminates ≥ `Trace` (0.001) on every fixture row and strictly once the remembered pattern ages past its prune point (disc 0.0188 vs 0.0000 at 30 stale / age 20 and 60); default = `Marker`. Consumer D-HOUSE-4 unblocked | -| D-TEH-3 | W2: calibration MATH → jc (ruling 4: compare, then lift or perfect in jc; crate copies deleted); `semantic_chunker` / `spiral_segment` decided by their falsifiers | jc / deepnsm-v2 / codec home | **Math half Shipped 2026-09-02** — new `jc::drift` (`reencode_drift` / `reencode_batch` / `delta_summary`) and `jc::quorum` (`pairwise_agreement_u8` / `QuorumLevel` / `cronbach_report`); the lift gate ran on distinguishing fixtures: cronbach = same estimator (LIFT; the `f32` copy loses a `1e7`-shifted fixture the `f64` form holds to `1e-9`), spearman = the retired copy ranked ties by position (PERFECT-IN-JC: 1.000 vs 0.948683 on `[1,2,2,3]`, indistinguishable on the tie-free fixtures its own tests used). Lab: `cronbach.rs` deleted; `reencode_safety` / `silu_correction` / `ground_truth::calibration` are glue over jc, x256 proof green through jc (14 tests). `semantic_chunker` / `spiral_segment` halves still Queued on their falsifiers | +| D-TEH-3 | W2: calibration MATH → jc (ruling 4: compare, then lift or perfect in jc; crate copies deleted); `semantic_chunker` / `spiral_segment` decided by their falsifiers | jc / deepnsm-v2 / codec home | **Shipped 2026-09-02/03 — all three halves closed.** Math: new `jc::drift` (`reencode_drift` / `reencode_batch` / `delta_summary`) and `jc::quorum` (`pairwise_agreement_u8` / `QuorumLevel` / `cronbach_report`); lift gate ran on distinguishing fixtures: cronbach = same estimator (LIFT; the `f32` copy loses a `1e7`-shifted fixture the `f64` form holds to `1e-9`), spearman = the retired copy ranked ties by position (PERFECT-IN-JC: 1.000 vs 0.948683 on `[1,2,2,3]`). `cronbach.rs` deleted; `reencode_safety` / `silu_correction` / `ground_truth::calibration` are glue over jc, x256 proof green (14 tests). **`semantic_chunker`: KILL** (§4c) — recall 0.000 at every threshold, confirmed a genuine mechanism null (not a harness artifact) via a non-committed positive-control diagnostic on the module's own adversarial fixture shape; stays LAB. **`spiral_segment`: KILL** (§4c) — fidelity clears r/rho >= 0.9980 on every u8 table but NOT on i8 (rho tops out at 0.9975, corrected 2026-09-03 per Codex review on #1144); compression fails on every table regardless, topping out at ~0.28x a u8 table (i.e. ~3.6x LARGER, not the claimed 51x), because real HDR/CDF table rows need ~114-143 segments to hit even a loose max_error; stays LAB, certification battery not scheduled | | D-TEH-4 | W3: M8 engine collapse with parity suite; cascade shapes and lens modules collapse | thinking-engine → the one engine | Queued (owns ENTROPY M8) | | D-TEH-5 | W4: residue deleted, crate renamed `thinking-lab` with a CI line; §2 rows closed; TD paid | workspace | Queued — closes the chapter | diff --git a/crates/thinking-engine/examples/chunker_falsifier.rs b/crates/thinking-engine/examples/chunker_falsifier.rs new file mode 100644 index 000000000..f8425f66f --- /dev/null +++ b/crates/thinking-engine/examples/chunker_falsifier.rs @@ -0,0 +1,289 @@ +//! chunker_falsifier — the D-TEH-3 fate probe for `semantic_chunker`. +//! +//! Plan `thinking-engine-harvest-closure-v1` §1c: `semantic_chunker` goes to +//! deepnsm-v2 ONLY if its falsifier passes ("boundaries vs a gold sentence +//! split; else stays LAB"). This is that falsifier, on REAL data: +//! +//! - text: the tier-1..4 calibration corpus of `jina_v5_ground_truth.rs` +//! (Rule 23 — no synthetic sentences); +//! - tokens → centroids: the Jina v5 (Qwen3) tokenizer + the baked +//! `jina-v5-codebook/codebook_index.u16` (151 936 entries); +//! - engine: the baked `jina-v5-codebook/distance_table_256x256.u8`. +//! +//! Three arms, PRE-REGISTERED before the first run: +//! +//! - CAN-FIRE: every ordered pair of sentences from DIFFERENT corpus pairs is +//! concatenated; the gold boundary is the token seam. recall@tol = fraction +//! of passages with a detected boundary within ±`SEAM_TOL` tokens of the +//! seam. +//! - NULL: the same passages with their centroid order shuffled +//! (`NULL_PERMS` deterministic SplitMix64 permutations) — the seam no longer +//! exists, so this is what "a boundary landed near the seam by chance" +//! looks like. Reported as p95 of recall@tol over permutations. +//! - SILENCE: the two sentences of each tier-1 / tier-2 pair (same topic), +//! both orders; a boundary here is a false split. +//! +//! PASS (per threshold, the SAME threshold for all three arms): +//! recall@tol ≥ 0.75 AND recall@tol ≥ null_p95 + 0.15 +//! AND false splits ≤ 2 of the 8 coherent passages. +//! KILL otherwise → the chunker stays LAB (not ported). +//! +//! A 4th, committed arm — POSITIVE CONTROL: an all-zero recall on the three +//! arms above is exactly the shape a broken harness (wrong table, wrong +//! tokenizer, a wiring bug) would also produce, so a null result is not +//! trustworthy on its own (CLAUDE.md's falsifiability rule: "a null result is +//! a claim about the measurement apparatus until proven otherwise"). This arm +//! reproduces the module's OWN adversarial positive-control shape from +//! `semantic_chunker::tests::detects_boundary_between_topics` — two +//! maximally-separated synthetic centroid clusters (corners 0-4 vs 250-254 of +//! the 256-centroid space) — against the SAME real table the falsifier uses, +//! and reports its boundary count at every threshold. If this control ALSO +//! returns zero, the all-zero falsifier result is a genuine mechanism null, +//! not a harness artifact; if it returns boundaries where the falsifier did +//! not, the harness (not the mechanism) is the thing to investigate. +//! +//! Usage: +//! JINA_V5_TOKENIZER=/path/to/tokenizer.json \ +//! cargo run --release --manifest-path crates/thinking-engine/Cargo.toml \ +//! --example chunker_falsifier + +use std::path::Path; + +use thinking_engine::codebook_index::CodebookIndex; +use thinking_engine::engine::ThinkingEngine; +use thinking_engine::semantic_chunker::{find_boundaries, ChunkerConfig}; + +const TABLE: &str = "crates/thinking-engine/data/jina-v5-codebook/distance_table_256x256.u8"; +const CODEBOOK: &str = "crates/thinking-engine/data/jina-v5-codebook/codebook_index.u16"; +const N: usize = 256; + +/// ±tokens around the seam that count as "found it". Two chunker steps. +const SEAM_TOL: usize = 4; +const NULL_PERMS: usize = 20; +const THRESHOLDS: &[f32] = &[0.3, 0.45, 0.6]; + +/// The real corpus (jina_v5_ground_truth.rs), grouped by pair; pairs 0..=3 +/// are tier 1–2 (same topic within the pair), 4 is tier 3, 5–6 are tier 4. +const CORPUS: &[(&str, &str)] = &[ + ( + "The wound is the place where the light enters you", + "Where there is ruin there is hope for a treasure", + ), + ( + "A federal judge in New York ruled the surveillance program unconstitutional", + "A US court declared the mass surveillance scheme violated the constitution", + ), + ( + "Palantir built Gotham for intelligence agencies to map human networks", + "Edward Snowden revealed the NSA collected phone metadata of millions of Americans", + ), + ( + "Amyloid plaques accumulate in the brains of Alzheimer patients", + "Tau protein tangles disrupt neural communication in neurodegenerative disease", + ), + ( + "Newton showed that gravity follows an inverse square law", + "Quantum entanglement allows particles to share states across arbitrary distances", + ), + ( + "You are not a drop in the ocean you are the entire ocean in a drop", + "TCP uses a three-way handshake to establish a reliable connection between hosts", + ), + ( + "CRISPR-Cas9 enables precise editing of genomic sequences at targeted loci", + "Bach composed the Well-Tempered Clavier as an exploration of all major and minor keys", + ), +]; +const COHERENT_PAIRS: usize = 4; + +struct SplitMix64(u64); +impl SplitMix64 { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn shuffle(&mut self, v: &mut [T]) { + for i in (1..v.len()).rev() { + let j = (self.next() % (i as u64 + 1)) as usize; + v.swap(i, j); + } + } +} + +fn config(threshold: f32) -> ChunkerConfig { + ChunkerConfig { + window_size: 8, + step_size: 2, + boundary_threshold: threshold, + min_chunk_tokens: 4, + max_chunk_tokens: 256, + top_k: 5, + max_cycles: 10, + } +} + +fn near_seam(engine: &mut ThinkingEngine, cents: &[u16], seam: usize, cfg: &ChunkerConfig) -> bool { + find_boundaries(engine, cents, cfg) + .iter() + .any(|b| b.position.abs_diff(seam) <= SEAM_TOL) +} + +fn main() { + let tok_path = std::env::var("JINA_V5_TOKENIZER") + .unwrap_or_else(|_| "/tmp/jina-v5-tokenizer.json".to_string()); + let tokenizer = match tokenizers::Tokenizer::from_file(&tok_path) { + Ok(t) => t, + Err(e) => { + eprintln!("tokenizer not loadable at {tok_path}: {e}"); + eprintln!("set JINA_V5_TOKENIZER to the Jina v5 (Qwen3) tokenizer.json"); + std::process::exit(2); + } + }; + let codebook = CodebookIndex::load(Path::new(CODEBOOK), N as u16, "jina-v5".into()) + .expect("codebook index"); + let table = std::fs::read(TABLE).expect("table"); + assert_eq!(table.len(), N * N); + let mut engine = ThinkingEngine::new(table); + + // Tokenize once (no special tokens, so a concatenation's seam is len(a)). + let sentences: Vec<(usize, Vec)> = CORPUS + .iter() + .enumerate() + .flat_map(|(pair, (a, b))| [(pair, *a), (pair, *b)]) + .map(|(pair, text)| { + let ids = tokenizer + .encode(text, false) + .expect("tokenize") + .get_ids() + .to_vec(); + (pair, codebook.lookup_many(&ids)) + }) + .collect(); + let (min_len, max_len) = sentences.iter().fold((usize::MAX, 0), |(lo, hi), (_, c)| { + (lo.min(c.len()), hi.max(c.len())) + }); + println!( + "corpus: {} sentences, {}..{} tokens each, codebook {} tokens / {} centroids used", + sentences.len(), + min_len, + max_len, + codebook.len(), + codebook.unique_centroids() + ); + + // Passages. + let mut switch: Vec<(Vec, usize)> = Vec::new(); // (centroids, seam) + for (i, (pa, ca)) in sentences.iter().enumerate() { + for (j, (pb, cb)) in sentences.iter().enumerate() { + if i != j && pa != pb { + let mut v = ca.clone(); + v.extend_from_slice(cb); + switch.push((v, ca.len())); + } + } + } + let mut coherent: Vec<(Vec, usize)> = Vec::new(); + for pair in 0..COHERENT_PAIRS { + let (_, a) = &sentences[2 * pair]; + let (_, b) = &sentences[2 * pair + 1]; + for (x, y) in [(a, b), (b, a)] { + let mut v = x.clone(); + v.extend_from_slice(y); + coherent.push((v, x.len())); + } + } + println!( + "passages: {} cross-topic (can-fire), {} same-topic (silence); seam tolerance ±{SEAM_TOL}, {NULL_PERMS} null permutations\n", + switch.len(), + coherent.len() + ); + + println!( + "{:>9} | {:>9} | {:>8} | {:>12} | {:>9} | verdict", + "threshold", "recall", "null p95", "false splits", "bnd/pass" + ); + let mut any_pass = false; + for &t in THRESHOLDS { + let cfg = config(t); + // Can-fire. + let hits = switch + .iter() + .filter(|(c, s)| near_seam(&mut engine, c, *s, &cfg)) + .count(); + let recall = hits as f64 / switch.len() as f64; + let mean_bnd = switch + .iter() + .map(|(c, _)| find_boundaries(&mut engine, c, &cfg).len()) + .sum::() as f64 + / switch.len() as f64; + // Null: shuffle each passage's centroid order, keep the nominal seam. + let mut rng = SplitMix64(0x9E37_79B9_7F4A_7C15); + let mut null_recalls: Vec = (0..NULL_PERMS) + .map(|_| { + let h = switch + .iter() + .filter(|(c, s)| { + let mut v = c.clone(); + rng.shuffle(&mut v); + near_seam(&mut engine, &v, *s, &cfg) + }) + .count(); + h as f64 / switch.len() as f64 + }) + .collect(); + null_recalls.sort_by(|a, b| a.total_cmp(b)); + let p95 = null_recalls[((NULL_PERMS as f64 * 0.95).ceil() as usize).min(NULL_PERMS) - 1]; + // Silence. + let false_splits = coherent + .iter() + .filter(|(c, _)| !find_boundaries(&mut engine, c, &cfg).is_empty()) + .count(); + let pass = recall >= 0.75 && recall >= p95 + 0.15 && false_splits <= 2; + any_pass |= pass; + println!( + "{:>9.2} | {:>9.3} | {:>8.3} | {:>7} of {:>2} | {:>9.2} | {}", + t, + recall, + p95, + false_splits, + coherent.len(), + mean_bnd, + if pass { "PASS" } else { "kill" } + ); + } + println!( + "\nVERDICT: {} — pre-registered: recall ≥ 0.75 AND recall ≥ null p95 + 0.15 AND false splits ≤ 2, at one threshold", + if any_pass { "PASS (port-eligible)" } else { "KILL (stays LAB)" } + ); + + // POSITIVE CONTROL — the module's own adversarial shape (two maximally + // separated synthetic centroid clusters), against the same real table. + // See the module doc comment above for why this arm exists. + let mut corners: Vec = Vec::with_capacity(48); + for i in 0..24u16 { + corners.push(i % 5); + } + for i in 0..24u16 { + corners.push(250 + i % 5); + } + println!("\n{:>9} | {:>18} | note", "threshold", "control boundaries"); + let mut control_ever_fires = false; + for &t in THRESHOLDS { + let cfg = config(t); + let n = find_boundaries(&mut engine, &corners, &cfg).len(); + control_ever_fires |= n > 0; + println!("{:>9.2} | {:>18} |", t, n); + } + println!( + "\nPOSITIVE CONTROL: {} — synthetic corners (centroids 0-4 vs 250-254, the module's own `detects_boundary_between_topics` shape) {} boundaries on this table", + if control_ever_fires { + "FIRES (the falsifier's all-zero result is NOT a mechanism null — investigate the harness)" + } else { + "STAYS AT ZERO TOO — the falsifier's all-zero result is a genuine mechanism null, not a harness artifact" + }, + if control_ever_fires { "produces" } else { "produces zero" } + ); +} diff --git a/crates/thinking-engine/examples/spiral_gate_probe.rs b/crates/thinking-engine/examples/spiral_gate_probe.rs new file mode 100644 index 000000000..06699a247 --- /dev/null +++ b/crates/thinking-engine/examples/spiral_gate_probe.rs @@ -0,0 +1,206 @@ +//! spiral_gate_probe — the D-TEH-3 fate probe for `spiral_segment`. +//! +//! Plan `thinking-engine-harvest-closure-v1` §1c: `spiral_segment` (8 bytes +//! per row: anfang / ende / stride / gamma, "51× compression") is promoted to +//! a codec home only through the certification battery. The full battery +//! needs the F32 cosine matrix re-derived from the model source +//! (certification-officer, Rule 7) — which is not on disk here. This probe +//! is the GATE in front of that expensive step: it fits the codec to the +//! four REAL baked tables in the tree and measures whether it can clear the +//! encoding-ecosystem floor at all. A KILL here is final (a codec that cannot +//! preserve a baked 256×256 table will not preserve its F32 parent); a PASS +//! unblocks the battery, it is not a certification. +//! +//! Tables (all real, all committed): jina-v3 u8 CDF, bge-m3 u8 CDF, +//! reranker u8 CDF, jina-v5 u8 CDF and jina-v5 i8 direct (`round(cos·127)`). +//! Values are mapped to [0, 1] (u8/255) or [−1, 1] (i8/127) and handed to +//! the codec as BF16 exactly as `SpiralTable::encode` does; the reference for +//! every metric is the ORIGINAL table value, so BF16 rounding is charged to +//! the codec (it stores BF16). The BF16-rounded input is reported as a second +//! reference so the fit error can be told apart from the storage error. +//! +//! PRE-REGISTERED (encoding-ecosystem.md: "any encoding below the naive u8 +//! floor is worse than doing nothing"; the bgz-hhtl-d gate is Pearson ≥ +//! 0.9980): PASS if for SOME `max_error` on EVERY table +//! Pearson r ≥ 0.9980 AND Spearman ρ ≥ 0.9980 AND +//! bytes(spiral) ≤ bytes(u8 table) / 2 (it must beat what it replaces) +//! KILL otherwise → stays LAB, the 51× claim recorded as measured-false. +//! +//! Usage: +//! cargo run --release --manifest-path crates/thinking-engine/Cargo.toml \ +//! --example spiral_gate_probe + +use std::collections::HashSet; + +use bgz_tensor::stacked_n::{bf16_to_f32, f32_to_bf16}; +use jc::reliability::{cronbach_alpha, pearson, spearman}; +use thinking_engine::spiral_segment::SpiralRow; + +const N: usize = 256; +const MAX_ERRORS: &[f32] = &[0.005, 0.01, 0.02, 0.05]; +const R_FLOOR: f64 = 0.9980; +const RHO_FLOOR: f64 = 0.9980; +const MIN_RATIO_VS_U8: f64 = 2.0; + +enum Kind { + U8, + I8, +} + +const TABLES: &[(&str, &str, Kind)] = &[ + ( + "jina-v3 u8", + "crates/thinking-engine/data/jina-v3-hdr/distance_table_256x256.u8", + Kind::U8, + ), + ( + "bge-m3 u8", + "crates/thinking-engine/data/bge-m3-hdr/distance_table_256x256.u8", + Kind::U8, + ), + ( + "reranker u8", + "crates/thinking-engine/data/jina-reranker-v3-BF16-hdr/distance_table_256x256.u8", + Kind::U8, + ), + ( + "jina-v5 u8", + "crates/thinking-engine/data/jina-v5-codebook/distance_table_256x256.u8", + Kind::U8, + ), + ( + "jina-v5 i8", + "crates/thinking-engine/data/jina-v5-codebook/distance_table_256x256.i8", + Kind::I8, + ), +]; + +struct Row { + /// Pearson r / Spearman ρ / Cronbach α vs the ORIGINAL table values. + r: f64, + rho: f64, + alpha: f64, + /// Same three vs the BF16-rounded input (isolates the fit from storage). + r_bf16: f64, + rho_bf16: f64, + /// Spiral bytes for the whole table, and the two baselines. + bytes: usize, + ratio_vs_u8: f64, + ratio_vs_bf16: f64, + avg_segments: f64, + max_abs_err: f64, +} + +fn load(path: &str, kind: &Kind) -> Vec { + let raw = std::fs::read(path).unwrap_or_else(|e| panic!("{path}: {e}")); + assert_eq!(raw.len(), N * N, "{path}: not a 256×256 table"); + match kind { + Kind::U8 => raw.iter().map(|&b| f32::from(b) / 255.0).collect(), + Kind::I8 => raw.iter().map(|&b| f32::from(b as i8) / 127.0).collect(), + } +} + +fn probe(table: &[f32], max_error: f32) -> Row { + let mut reference = Vec::with_capacity(N * (N - 1)); + let mut ref_bf16 = Vec::with_capacity(N * (N - 1)); + let mut decoded = Vec::with_capacity(N * (N - 1)); + let mut bytes = 0usize; + let mut segments = 0usize; + let mut max_abs_err = 0.0f64; + for i in 0..N { + let row: Vec = (0..N) + .filter(|&j| j != i) + .map(|j| table[i * N + j]) + .collect(); + let enc = SpiralRow::encode(&row, table[i * N + i], max_error); + let dec = enc.decode(N - 1); + bytes += enc.byte_size(); + segments += enc.segments.len(); + for (k, &v) in row.iter().enumerate() { + let d = f64::from(dec[k]); + let orig = f64::from(v); + reference.push(orig); + ref_bf16.push(f64::from(bf16_to_f32(f32_to_bf16(v)))); + decoded.push(d); + max_abs_err = max_abs_err.max((d - orig).abs()); + } + } + let nan = f64::NAN; + Row { + r: pearson(&reference, &decoded).unwrap_or(nan), + rho: spearman(&reference, &decoded).unwrap_or(nan), + alpha: cronbach_alpha(&[reference.clone(), decoded.clone()]).unwrap_or(nan), + r_bf16: pearson(&ref_bf16, &decoded).unwrap_or(nan), + rho_bf16: spearman(&ref_bf16, &decoded).unwrap_or(nan), + bytes, + ratio_vs_u8: (N * N) as f64 / bytes as f64, + ratio_vs_bf16: (N * N * 2) as f64 / bytes as f64, + avg_segments: segments as f64 / N as f64, + max_abs_err, + } +} + +fn main() { + let dedup_set: HashSet<&str> = TABLES.iter().map(|(n, _, _)| *n).collect(); + assert_eq!(dedup_set.len(), TABLES.len()); + println!( + "gate: r ≥ {R_FLOOR}, ρ ≥ {RHO_FLOOR}, spiral bytes ≤ u8 bytes / {MIN_RATIO_VS_U8} — on EVERY table at ONE max_error\n" + ); + println!( + "{:>12} | {:>9} | {:>7} | {:>7} | {:>7} | {:>7} | {:>7} | {:>7} | {:>6} | {:>8} | {:>7} | {:>9}", + "table", + "max_error", + "r", + "rho", + "alpha", + "r_bf16", + "rho_bf16", + "x u8", + "x bf16", + "seg/row", + "max|e|", + "bytes" + ); + // pass_at[k] = every table clears the gate at MAX_ERRORS[k]. + let mut pass_at = vec![true; MAX_ERRORS.len()]; + for (name, path, kind) in TABLES { + let table = load(path, kind); + for (k, &me) in MAX_ERRORS.iter().enumerate() { + let t0 = std::time::Instant::now(); + let row = probe(&table, me); + let ok = row.r >= R_FLOOR && row.rho >= RHO_FLOOR && row.ratio_vs_u8 >= MIN_RATIO_VS_U8; + pass_at[k] &= ok; + println!( + "{:>12} | {:>9.3} | {:>7.4} | {:>7.4} | {:>7.4} | {:>7.4} | {:>7.4} | {:>7.2} | {:>6.2} | {:>8.2} | {:>7.4} | {:>9} {} ({:.1}s)", + name, + me, + row.r, + row.rho, + row.alpha, + row.r_bf16, + row.rho_bf16, + row.ratio_vs_u8, + row.ratio_vs_bf16, + row.avg_segments, + row.max_abs_err, + row.bytes, + if ok { "ok" } else { "--" }, + t0.elapsed().as_secs_f64() + ); + } + } + let passing: Vec = MAX_ERRORS + .iter() + .zip(&pass_at) + .filter(|(_, p)| **p) + .map(|(m, _)| *m) + .collect(); + println!( + "\nVERDICT: {}", + if passing.is_empty() { + "KILL (stays LAB) — no max_error clears the gate on every table".to_string() + } else { + format!("PASS (battery-eligible) at max_error {passing:?}") + } + ); +} diff --git a/crates/thinking-engine/src/cognitive_stack.rs b/crates/thinking-engine/src/cognitive_stack.rs index 74ab753de..a8e38dda5 100644 --- a/crates/thinking-engine/src/cognitive_stack.rs +++ b/crates/thinking-engine/src/cognitive_stack.rs @@ -238,6 +238,12 @@ pub struct MetaAssessment { pub calibration_error: f32, } +impl Default for MetaCognition { + fn default() -> Self { + Self::new() + } +} + impl MetaCognition { pub fn new() -> Self { Self { diff --git a/crates/thinking-engine/src/dto.rs b/crates/thinking-engine/src/dto.rs index 4d4c47b03..b3e011841 100644 --- a/crates/thinking-engine/src/dto.rs +++ b/crates/thinking-engine/src/dto.rs @@ -184,6 +184,7 @@ impl ThoughtStruct { /// Structure-of-Arrays for cognitive search across many thoughts. /// Same principle as NeuronIndex: AoS for API, SoA for search. +#[derive(Default)] pub struct ThoughtIndex { pub codebook_index: Vec, pub energy: Vec, diff --git a/crates/thinking-engine/src/qualia.rs b/crates/thinking-engine/src/qualia.rs index 407b29d39..fbb6157c3 100644 --- a/crates/thinking-engine/src/qualia.rs +++ b/crates/thinking-engine/src/qualia.rs @@ -452,9 +452,9 @@ impl Qualia17D { /// Each f32 [0,1] dim maps to i8 [-127, 127] via: (dim - 0.5) * 254. pub fn to_voice_channels(&self) -> [i8; 16] { let mut channels = [0i8; 16]; - for i in 0..16 { - let v = self.dims[i]; // skip dim 16 (integration) for 16-channel fit - channels[i] = ((v - 0.5) * 254.0).clamp(-127.0, 127.0) as i8; + // skip dim 16 (integration) for 16-channel fit + for (c, &v) in channels.iter_mut().zip(self.dims.iter()) { + *c = ((v - 0.5) * 254.0).clamp(-127.0, 127.0) as i8; } channels } @@ -545,91 +545,53 @@ impl Qualia17D { match family { "emberglow" => { // Warm: boost 800-2500 Hz (formant region) - for i in 4..=10 { - w[i] = 1.3; - } + w[4..=10].fill(1.3); } "woodwarm" => { // Grounded: boost bass + low-mid - for i in 0..=6 { - w[i] = 1.2; - } - for i in 15..=20 { - w[i] = 0.85; - } + w[0..=6].fill(1.2); + w[15..=20].fill(0.85); } "steelwind" => { // Sharp: boost presence (2-5 kHz) - for i in 10..=14 { - w[i] = 1.4; - } - for i in 0..=3 { - w[i] = 0.8; - } + w[10..=14].fill(1.4); + w[0..=3].fill(0.8); } "oceandrift" => { // Flowing: gentle mid, soft treble - for i in 6..=12 { - w[i] = 1.1; - } - for i in 16..=20 { - w[i] = 0.9; - } + w[6..=12].fill(1.1); + w[16..=20].fill(0.9); } "frostbite" => { // Cold: boost treble, cut warmth - for i in 14..=20 { - w[i] = 1.3; - } - for i in 4..=8 { - w[i] = 0.7; - } + w[14..=20].fill(1.3); + w[4..=8].fill(0.7); } "sunburst" => { // Bright: broadband boost, emphasis on harmonics - for i in 0..=20 { - w[i] = 1.1; - } - for i in 8..=14 { - w[i] = 1.3; - } + w[0..=20].fill(1.1); + w[8..=14].fill(1.3); } "nightshade" => { // Dark: deep bass, soft everything else - for i in 0..=4 { - w[i] = 1.4; - } - for i in 10..=20 { - w[i] = 0.7; - } + w[0..=4].fill(1.4); + w[10..=20].fill(0.7); } "thornrose" => { // Tense: mid emphasis + presence peak - for i in 6..=8 { - w[i] = 1.3; - } + w[6..=8].fill(1.3); w[13] = 1.4; // sibilance spike } "velvetdusk" => { // Soft: gentle roll-off, warm low-mid - for i in 2..=8 { - w[i] = 1.15; - } - for i in 14..=20 { - w[i] = 0.8; - } + w[2..=8].fill(1.15); + w[14..=20].fill(0.8); } "stormbreak" => { // Aggressive: mid-scoop + treble + bass - for i in 0..=3 { - w[i] = 1.3; - } - for i in 6..=10 { - w[i] = 0.8; - } // mid scoop - for i in 14..=20 { - w[i] = 1.3; - } + w[0..=3].fill(1.3); + w[6..=10].fill(0.8); // mid scoop + w[14..=20].fill(1.3); } _ => {} // neutral } diff --git a/crates/thinking-engine/src/signed_domino.rs b/crates/thinking-engine/src/signed_domino.rs index 5e93014b2..2c6bb760c 100644 --- a/crates/thinking-engine/src/signed_domino.rs +++ b/crates/thinking-engine/src/signed_domino.rs @@ -72,11 +72,10 @@ impl<'a> SignedDominoCascade<'a> { } let row = &table[q_idx as usize * n..(q_idx as usize + 1) * n]; - for j in 0..n { + for (j, &val) in row.iter().enumerate() { if j == q_idx as usize { continue; } - let val = row[j]; // Sign IS the gate decision if val > 0 { diff --git a/crates/thinking-engine/src/tensor_bridge.rs b/crates/thinking-engine/src/tensor_bridge.rs index 367a5cfb5..ed7239e4d 100644 --- a/crates/thinking-engine/src/tensor_bridge.rs +++ b/crates/thinking-engine/src/tensor_bridge.rs @@ -121,6 +121,12 @@ impl EmbeddingBatch { } /// Pairwise cosine similarity matrix. + // Symmetric fill: each iteration writes matrix[i][j] AND matrix[j][i] — + // two different rows of `matrix` alongside two different, unrelated + // elements of `self.embeddings` — so there's no single iterator over one + // collection that expresses this without an equivalent index dance + // (clippy's own suggested rewrite only reaches row `i`, not row `j`). + #[allow(clippy::needless_range_loop)] pub fn pairwise_cosines(&self) -> Vec> { let n = self.embeddings.len(); let mut matrix = vec![vec![0.0f32; n]; n]; diff --git a/crates/thinking-engine/src/world_model.rs b/crates/thinking-engine/src/world_model.rs index dba934475..01a194b7a 100644 --- a/crates/thinking-engine/src/world_model.rs +++ b/crates/thinking-engine/src/world_model.rs @@ -149,22 +149,39 @@ pub struct WorldModelDto { pub context_state: ContextState, } +/// Per-cycle scalar signals fed into [`WorldModelDto::from_engine_state`], +/// grouped so the constructor takes the three stateful objects (`agent` / +/// `field` / `qualia`) plus one bundle instead of ten positional args. +#[derive(Clone, Debug)] +pub struct ThoughtSignals { + pub dissonance: f32, + pub free_energy: f32, + pub lens_agreement: f32, + pub spo_count: u16, + pub calibration_error: f32, + /// The thought's ghost-prior summary (planner `GhostPrior`): active count + /// and the dominant echo. The agent no longer owns the field (D-TEH-2). + pub trace_count: u16, + pub dominant_trace: Option, +} + impl WorldModelDto { /// Build from the thinking engine's current state. pub fn from_engine_state( agent: &crate::persona::Agent, field: &crate::superposition::SuperpositionField, qualia: &crate::qualia::Qualia17D, - dissonance: f32, - free_energy: f32, - lens_agreement: f32, - spo_count: u16, - calibration_error: f32, - // The thought's ghost-prior summary (planner `GhostPrior`): active count - // and the dominant echo. The agent no longer owns the field (D-TEH-2). - trace_count: u16, - dominant_trace: Option, + signals: ThoughtSignals, ) -> Self { + let ThoughtSignals { + dissonance, + free_energy, + lens_agreement, + spo_count, + calibration_error, + trace_count, + dominant_trace, + } = signals; let hdr = HdrResonance::new( lens_agreement, 1.0 - dissonance,