Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
## 2026-09-02 — E-THE-LIFT-GATE-FOUND-A-TIE-BLIND-SPEARMAN-1 — the D-TEH-3 comparison on a distinguishing fixture separated a same-formula copy from a wrong-estimator copy, and the tie-free fixtures the lab had used could not have

**Status:** FINDING (measured, tests committed in `crates/jc/src/reliability.rs`). **Confidence:** High.

**What the ruling asked.** `E-JC-IS-THE-HOME-OF-ALL-CALIBRATED-MATH-1`: a calibration routine outside jc is LIFTED if correct, or the jc version is PERFECTED and the copy dies — decided on a fixture that can distinguish the two implementations. D-TEH-3 ran that gate on the thinking-engine battery. Four verdicts, two of them not the one a code read would have given.

**Cronbach α — LIFT, with a precision caveat the fixture had to expose.** `thinking_engine::cronbach::cronbach_alpha` and `jc::reliability::cronbach_alpha` are the same estimator (population variances, `k/(k−1)·(1 − Σσ²/σ²_total)`); on the known-value fixture they agree to `1e-5` (0.984615). That agreement proves nothing by itself, so the same fixture was shifted by `1e7`: α is affine-invariant in exact arithmetic and the `f64` estimate does not move (`< 1e-9`), while the retired `f32` form loses the fixture outright (its mean rounds to the nearest unit at that magnitude and the deviations carry no information). The copy dies; jc's `f64`/`Option` form is the survivor, not because it is "newer" but because the fixture showed which one survives the data.

**Spearman ρ — PERFECT-IN-JC, and the lab's own tests were structurally blind to it.** `ground_truth::spearman_rank_correlation` ranked ties by sort POSITION (index order), not by average rank. On tie-free data it agrees with jc to `1e-6` — and every one of its three retired unit tests was tie-free. On `x=[1,2,3,4]`, `y=[1,2,2,3]` it returns ρ = 1.000 (a "perfect monotone match"); the average-rank form returns the textbook 0.948683. A calibration battery that compares baked `u8` lens distances against ground-truth cosines will hit ties constantly (256 levels over thousands of pairs), so the retired form was systematically optimistic exactly where it was used. jc already carried the tie correction; nothing was added to it except the comparison test that keeps the retired form on record.

**Re-encode drift and the correction-delta summary — LIFTED as-is into `jc::drift`.** The convergence statistic (`|e_i − e_{i−1}| < 1e-15` against the ORIGINAL value, tail-filled history, `safe = converged_at < max_iterations`) and the delta battery (mean / mean|δ| / max|δ| / population σ / two threshold fractions) were correct; what changed is their shape — the codec choice (BF16, γ+φ, the full chain) is now a round-trip closure the lab passes in, and the two cut-offs (0.01 / 0.1) are parameters the lab names (`MATERIAL_CORRECTION` / `LARGE_CORRECTION`) instead of constants buried in the statistic. The x256 re-encode proof (14 lab tests) runs unchanged through jc. **The lens quorum** (per-pair `1 − σ/σ_max` over `u8` tables, the 230/179/128 bands, the per-subject Cronbach report) went to `jc::quorum` the same way; it never was an α per pair and its docs now say so in the type name.

**The generalizable point.** "Agrees on the existing tests" was true for both copies and meant nothing: the tests were the fixtures on which the implementations coincide. The gate had to be run on the input class each estimator would actually see (large offsets for `f32`, ties for ranks). A lift decision made from a code read would have kept the tie-blind Spearman — it looks like the textbook formula.

**Still outside jc, recorded rather than swept.** `ndarray::hpc::reliability` (tie-aware, `f64`, returns `0.0` on degeneracy) and `perturbation-sim::stats` (a deliberate zero-dep mirror of it) carry the same four estimators. Both are correct in the sense above; neither is jc. Filed as `TD-RELIABILITY-COPIES-OUTSIDE-JC-1` — the ruling's own text names the ndarray direction ("once ndarray is proven bit-exact the same math is re-importable from jc"), which is a deliberate PR with a bit-exactness gate, not a drive-by.

Refs: `E-JC-IS-THE-HOME-OF-ALL-CALIBRATED-MATH-1`; `thinking-engine-harvest-closure-v1` §1d / §3 W2 / §5 D-TEH-3; `crates/jc/src/{reliability,drift,quorum}.rs`; the falsifiability rule (CLAUDE.md § The falsifiability rule).

## 2026-09-02 — E-THE-CALIBRATION-GATE-REVERSED-THE-DECLARED-FLOOR-1 — D-TEH-2: the ghost prior landed in the planner, and the pre-registered gate overruled the floor its author had declared

**Status:** FINDING (measured, `crates/lance-graph-planner/src/nars/ghost_prior.rs`, test `calibration_gate_picks_the_default_floor`). **Confidence:** High on the fixture; the fixture is one recurrence shape (256 atoms, a 3-atom pattern, 0–30 stale patterns, ages 0–60).
Expand Down
10 changes: 10 additions & 0 deletions .claude/board/LATEST_STATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
## 2026-09-02 — branch (D-TEH-3, after #1142 merged): calibration math lifted into jc — INVENTORY DELTA

- MERGED #1142 (D-TEH-2, `3c5f040`): no contract type; planner `nars/ghost_prior.rs` + lab-crate edits + `weather-poc.yml` sibling checkout (see PR_ARC).
- ADDED `jc::drift` — `ReencodeDrift` / `reencode_drift(value, max_iterations, codec, round_trip: FnMut(f32) -> f32)` / `DriftBatch` / `reencode_batch` (the re-encode convergence statistic, codec-agnostic), `DeltaSummary` / `delta_summary(deltas, material_threshold, large_threshold) -> Option` / `DeltaSummary::empty`, `CONVERGENCE_EPS`. 8 tests (identity converges at 1 with a filled tail; multiplicative drift never converges with a strictly growing history; damped codec converges late and the budget binds two-sided; zero budget proves nothing; batch worst/count; hand-computed summary; thresholds load-bearing both ways; empty / non-finite → `None`).
- ADDED `jc::quorum` — `pairwise_agreement_u8(tables, n) -> Option<Vec<u8>>` (per-pair `1 − σ/σ_max(k)`, `max_u8_variance(k)` = `255²·⌊k/2⌋⌈k/2⌉/k²` — the lifted source used the even-`k` ceiling for every `k`, so a maximally split 3-lens cell scored 15 instead of 0; perfected in jc per the ruling, Codex on the lift PR), `QuorumLevel { High, Medium, Low, Ambiguous }` with `HIGH_FLOOR`/`MEDIUM_FLOOR`/`LOW_FLOOR` = 230/179/128 and `needs_leaf_validation`, `CronbachReport` / `cronbach_report(items) -> Option` (α by delegation to `reliability::cronbach_alpha` + per-subject variances + the `mean + σ` disagreement count). 7 tests (coincidence → 255 off-diagonal too; odd-`k` `[0,0,255]` → 0 and `[0,0,128]` strictly between; 0-vs-255 → 0 and a 100-vs-150 pair → 205; degenerate → `None`; both sides of every band floor; the one disputed subject is the one flagged, with a mild subject above the mean but under `mean + σ` as the anti-vacuity for the disable; `None` where α is).
- ADDED `jc::reliability` tests only — the two D-TEH-3 lift-gate comparisons carrying the retired lab forms verbatim (`retired_cronbach_f32`, `retired_spearman_ordinal`): cronbach agrees on the known-value fixture and loses the `1e7`-shifted one; spearman agrees tie-free and diverges on one tie (1.000 vs 0.948683). No estimator changed. jc lib: 135/135.
- REMOVED (lab crate, excluded) `thinking_engine::cronbach` (`cronbach_alpha`, `variance_agreement_scores`, `QuorumLevel`, `CronbachResult`, `cronbach_analysis`) and `ground_truth::{spearman_rank_correlation, ranks}` with their tests. `reencode_safety.rs` is now glue over `jc::drift` (`pub type ReencodeSafety = jc::drift::ReencodeDrift`; the three codec wrappers pass round-trip closures; `test_reencode_batch` keeps its tuple shape; `test_zipper_offsets` + all 14 tests unchanged and green). `silu_correction.rs`: `pub use jc::drift::DeltaSummary as CorrectionStats`, `correction_stats -> Option<CorrectionStats>` is an adapter with the cut-offs named `MATERIAL_CORRECTION = 0.01f32` / `LARGE_CORRECTION = 0.1f32` (kept in the samples' precision and promoted with them, so a correction of exactly `0.1f32` stays "not more than" 0.1 — Codex); `None` = non-finite data (an invalid run), `Some(empty)` = genuinely empty (Codex: the first cut folded both into empty). `ground_truth::calibration::spearman_vs_ground_truth` calls `jc::reliability::spearman` (keeps its `0.0` fallback). `examples/certify_jina_v5_7lane.rs` calls `jc::reliability::cronbach_alpha` (`None` → `NaN`, which fails every `>=` verdict honestly). Lab `Cargo.toml` gains `jc = { path = "../jc" }`; lab lock adds `jc` and drops entries cargo no longer needs. Lab: `cargo check --lib --examples` clean except the pre-existing `tts_stream_hhtld` example break (bgz-tensor private fn, untouched, `TD-THINKING-ENGINE-EXCLUDED-DEBT-1`); lab tests for the three modules 23/23.
- NOT IN THIS DELTA: the `semantic_chunker` / `spiral_segment` halves of D-TEH-3 (falsifier-gated, separate); `silu` and `cosine_f32` stay in the lab (activation + vector glue, not calibrated math); `ndarray::hpc::reliability` and `perturbation-sim::stats` copies (`TD-RELIABILITY-COPIES-OUTSIDE-JC-1`).
- UNCHANGED: every contract type; every existing jc estimator's arithmetic, signature and semantics; the ALU artery files.

## 2026-09-02 — branch (D-TEH-2, after #1141 merged): ghost prior harvested into the planner — INVENTORY DELTA

- MERGED #1141 (D-HOUSE-1, `3102776`): no contract type; planner example `house_differential.rs` + plan/board records (see PR_ARC).
Expand Down
11 changes: 11 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@
> census §8.3 trap 10: read the body FIRST, then open for write — never
> inline both in one expression.

## 2026-09-02 — lance-graph branch `claude/medcare-rs-continue-6nhbxn` (D-TEH-3 PR, after #1142) — calibration MATH → jc; lab copies deleted

- **Added:** `crates/jc/src/drift.rs` (`ReencodeDrift`, `reencode_drift`, `DriftBatch`, `reencode_batch`, `DeltaSummary`, `delta_summary`, `CONVERGENCE_EPS`; 8 tests), `crates/jc/src/quorum.rs` (`pairwise_agreement_u8`, `max_u8_variance`, `QuorumLevel` + floors, `CronbachReport`, `cronbach_report`; 7 tests), two lift-gate tests in `crates/jc/src/reliability.rs`; `pub mod drift; pub mod quorum;` in jc `lib.rs`. EPIPHANIES `E-THE-LIFT-GATE-FOUND-A-TIE-BLIND-SPEARMAN-1`; TECH_DEBT `TD-RELIABILITY-COPIES-OUTSIDE-JC-1`.
- **Removed:** `crates/thinking-engine/src/cronbach.rs`; `spearman_rank_correlation` + `ranks` (+3 tests) from `ground_truth.rs`; the private statistic bodies in `reencode_safety.rs` and `silu_correction.rs`.
- **Changed (lab crate):** `Cargo.toml` (+`jc` path dep) and `Cargo.lock`; `lib.rs` (no `cronbach`); `reencode_safety.rs` (glue over `jc::drift`, tests verbatim); `silu_correction.rs` (`CorrectionStats` = `jc::drift::DeltaSummary`, named cut-offs); `ground_truth.rs` (calibration glue calls jc); `examples/certify_jina_v5_7lane.rs` (jc α, `Option` → `NaN`). Plan §5 D-TEH-3 → math half Shipped, §3 W2 result addendum; STATUS_BOARD D-TEH-3.
- **Measured:** cronbach — same formula, agree to `1e-5` on the 0.984615 fixture, `f32` copy loses the `1e7`-shifted fixture while `f64` moves `< 1e-9` → LIFT; spearman — retired form ranked ties by position, agrees `1e-6` tie-free, returns 1.000 vs jc 0.948683 on `[1,2,2,3]` → PERFECT-IN-JC (already there); drift + delta summary + quorum → lifted as-is with the codec / cut-offs turned into parameters. jc lib 135/135, new modules clippy `-D warnings` clean, fmt clean; lab 23/23 on the three modules, `check --lib --examples` clean but for the pre-existing `tts_stream_hhtld` break.
- **Review folded in (pre-merge, Codex P2 ×3):** odd-`k` quorum normalised by its own attainable variance (`max_u8_variance(k)`; `[0,0,255]` at `k = 3` scored 15 under the lifted even-`k` ceiling, now 0 — a defect the lift PERFECTED rather than carried); the lab cut-offs kept as `f32` and promoted with the samples (exactly `0.1f32` is not `> 0.1`); `correction_stats` returns `Option` — `None` for non-finite corrections, `Some(empty)` only for an empty set.
- **Locked:** jc is the only home of the four estimators inside lance-graph; the lab carries codec/cut-off choices, never a statistic; the retired forms stay in jc's tests as the comparison record.
- **Deferred:** D-TEH-3's `semantic_chunker` / `spiral_segment` halves (falsifier-gated); the ndarray / perturbation-sim copies (TD, deliberate PR with a bit-exactness gate); the pre-existing lab example break (D-TEH-5 / TD-THINKING-ENGINE-EXCLUDED-DEBT-1).
- **Confidence:** High.

## 2026-09-02 — lance-graph branch `claude/medcare-rs-continue-6nhbxn` (D-TEH-2 PR, after #1141) — ghost prior harvested into the planner; `ghosts.rs` deleted

- **Added:** `crates/lance-graph-planner/src/nars/ghost_prior.rs` — `GhostPrior`, `PriorFloor`, `Trace`, `calibration::{recurrence_fixture, discrimination}` (both `Option` below `FIXTURE_MIN_ATOMS` and above `FIXTURE_MAX_STALE_PATTERNS`), 15 tests (monotone decay to each floor with the inert cycle derived from the constant, decay-constant load-bearing both ways, two-sided free energy under both floors, the calibration gate on the declared default, anti-vacuity that the floors differ, prediction shape, imprint cap, echo rule, independence of two priors); `pub mod ghost_prior` + re-export in `nars/mod.rs`. EPIPHANIES `E-THE-CALIBRATION-GATE-REVERSED-THE-DECLARED-FLOOR-1`.
Expand Down
2 changes: 1 addition & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
| D-TEH-0 | census: live footprint measured (one required consumer via `bridge_gate`; one optional via `with-engine`), 51-file fate table, open-row reconciliation, four-wave closure, idea harvest, four rulings asked | **Shipped (plan-only, this PR)** |
| D-TEH-1 | W1: `bridge_gate` (seven items) → `lance_graph_contract::bridge_gate`; callcenter re-imports and drops the path dep; thinking-engine keeps a re-export shim | **Shipped 2026-09-02** (before: required dep, 6 crossing sites, dep-drop alone fails 6 × E0433; after: zero thinking-engine deps in callcenter metadata, contract 1303/1303, callcenter 156/156, driver default + `with-engine` green, ALU artery files byte-identical). `with-engine` re-point deferred by stop condition: D-TTV-1 not landed |
| D-TEH-2 | W2: ghost prior → planner `nars/ghost_prior.rs` over `WisdomMarker`, per-thought, two-sided falsifiers; crate `ghosts.rs` deleted | **Shipped 2026-09-02** — `GhostPrior`/`PriorFloor` in planner `nars/ghost_prior.rs`, 14 tests incl. the calibration gate (default floor = `Marker`, the gate reversed the first declaration); `ghosts.rs` + `think.rs` deleted; TD-GHOST-ECHO-DUP-1 resolved; consumer D-HOUSE-4 unblocked |
| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | Queued |
| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | **Math half Shipped 2026-09-02** — `jc::drift` (re-encode drift + delta summary) and `jc::quorum` (pairwise agreement, bands, Cronbach report) added; cronbach LIFTED (same formula; `f32` copy loses the `1e7`-shifted fixture), spearman PERFECTED-IN-JC (retired copy was tie-blind: 1.000 vs 0.9487 on one tie); lab `cronbach.rs` deleted, `reencode_safety` / `silu_correction` / `ground_truth` are glue over jc. Open: the `semantic_chunker` / `spiral_segment` falsifier halves |
| D-TEH-4 | W3: ENTROPY M8 engine collapse with dtype parity suite; 5 cascade shapes + 3 lens modules collapse | Queued |
| D-TEH-5 | W4: residue deleted, crate renamed `thinking-lab` with a `--manifest-path` CI line; §2 rows closed; `TD-THINKING-ENGINE-EXCLUDED-DEBT-1` paid | Queued — closes the chapter |

Expand Down
26 changes: 26 additions & 0 deletions .claude/board/TECH_DEBT.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,29 @@
## TD-RELIABILITY-COPIES-OUTSIDE-JC-1 (2026-09-02) — OPEN

**Two further copies of the Pearson / Spearman / Cronbach α / ICC battery live
outside `jc`.** Found while running the D-TEH-3 lift gate
(`E-THE-LIFT-GATE-FOUND-A-TIE-BLIND-SPEARMAN-1`):

- `ndarray::hpc::reliability` (`pearson`, `spearman`, `cronbach_alpha`,
`icc_a1`, `FidelityReport`) — tie-aware, `f64`, returns `0.0` on degenerate
input where jc returns `None`. The PRODUCTION copy; consumed by the
edge-codec fidelity work.
- `perturbation-sim::stats` — a self-described zero-dep mirror of the ndarray
copy so that crate's validation harness stays standalone.

Neither is wrong the way the retired thinking-engine Spearman was (both average
ranks over ties). They are still two more sources of truth for math the ruling
says has ONE home. The ruling itself names the direction for the first — *once
ndarray is proven bit-exact the same math is re-importable from jc* — so the
pay-down is a deliberate PR with a bit-exactness gate (same fixtures as jc's
lift-gate tests, plus the `0.0`-vs-`None` degeneracy contract decided
explicitly), never a drive-by delete. The perturbation-sim mirror follows
whatever the ndarray decision is.

Owner: whoever next touches `ndarray::hpc::reliability` or `perturbation-sim`.
Refs: `E-JC-IS-THE-HOME-OF-ALL-CALIBRATED-MATH-1`, `crates/jc/src/reliability.rs`
(lift-gate tests), `thinking-engine-harvest-closure-v1` §1d.

## TD-PILLAR11-SCIENTIFIC-LOOPS-BYPASS-NDARRAY-SIMD-1 (2026-09-02) — OPEN

**The debt is not missing SIMD support. It is scientific code bypassing the
Expand Down
Loading
Loading