diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index b8ec2e373..6033bcd75 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,12 @@ +## 2026-09-05 — PROBE-NXG-HIST-1: orchestrator-only, no worker spawned + +- **Why:** operator: "merge 1176 and start PROBE-NXG-HIST-1". #1176 merged (`a7520a15`). Plan §5 step 2 is a ~200-line probe with three pre-registered claims; briefing a worker costs more than writing it. +- **Column:** `data/tts-cascade/tts_real_output.wav` `|sample|` — real, in-tree, 94 572 rows. No facet fixture exists in-tree; noted in the probe header and E-NXG-17. +- **Result:** C1 PASS · C2 PASS · C3 PASS first run. Disable runs red-then-green: `AND3` for the bucket immediate (C1/C2/C3 all fail, exit 101); `partition_point(b <= v)` (C2 192 mismatches, exit 101); restored → green. +- **Correction found:** histogram-midpoint σ over-reads by 12 % on quantile buckets (heavy top bucket). Plan §3 room 3 regraded in E-NXG-17; entropy-on-requantiled-buckets is flat by construction (D-NXG-9 caveat). +- **Gates:** clippy `-D warnings` on the example clean (one `is_multiple_of` fix), fmt clean, append-only + citation-decay + plan-dids gates run locally, supersession index regenerated last. +- **Board:** E-NXG-17; STATUS_BOARD D-NXG-1/2/4 rows updated; plan §5 step 2 marked GREEN. + ## 2026-09-05 — Sonnet harvest fleet (4 readers + 7 PR sweepers), orchestrator-consolidated → `.claude/nexgen/` - **Why:** operator asked how HDR popcount-stacking, rolling floor, Prozentrang buckets, Mexican hat, Shannon, proprioception, EWA sandwich and sigker could combine with ternlog mask algebra for better thresholds; then asked for the last 50 lance-graph PRs + 25 OGAR + 25 ndarray swept for synergies, epiphanies written to the board, the raw agent output kept, and an expansion plan in the T0..T3 shape "27 rooms ahead". diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 997527c16..b85aea390 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,44 @@ +## 2026-09-05 — E-NXG-17 — PROBE-NXG-HIST-1 GREEN: the nested-mask histogram reproduces the partition-point rank on 94 572 real rows + +**Status:** FINDING (measured; `crates/lance-graph-planner/examples/probe_nxg_hist_1.rs`). +**Confidence:** High on C1–C3; the σ observation is a real correction to plan §3 room 3. + +The first gate of `nexgen-mask-histogram-thresholds-v1` §5 ran on a REAL +column — `|sample|` of `data/tts-cascade/tts_real_output.wav`, 94 572 rows of +16-bit speech, 1 478 mask words, B = 16 quantile bands. Three pre-registered +claims, all PASS on the first run: **C1** the 16 band masks are nested, the 16 +bucket masks are pairwise disjoint, and their popcounts sum to exactly N; +**C2** for every one of the 94 572 rows the bucket index read off the masks +equals `boundaries.partition_point(|b| b < v)` (0 mismatches; anti-vacuity +16/16 buckets non-empty, largest 5 925 ≈ N/16); **C3** +`mask_ternlog::(M_i, M_{i-1}, M_{i-1})` is bit-identical to +`mask_andnot(M_i, M_{i-1})` and `AND3` on the same operands differs at every i +with a non-empty lower band. Two disable runs, red-then-green: bucket via +`AND3` → C1 disjoint=false, sum=715 262, C2 88 659 mismatches, C3 false; +partition point `b <= v` → C2 192 mismatches (the rows sitting exactly on a +boundary — the off-by-one is real, not vacuous). E-NXG-1/2/3 are promoted from +PROPOSAL to FINDING for the structure; the cost claims stay unmeasured. + +**What the probe corrected.** Plan §3 room 3 said σ is "recoverable from the +histogram for free" via popcount-weighted bucket midpoints. Measured: σ_hist = +4 244.9 vs σ_direct = 3 785.8, ratio 1.121. Quantile buckets are equal-mass, +not equal-width, so the top bucket (9 299..20 634) is a heavy tail whose +midpoint over-weights it. Room 3 is regraded: σ from the histogram needs +per-bucket means (one more accumulator per bucket), not midpoints, and stays an +observation until that variant is measured. The entropy preview +(`thought_atoms::normalized_entropy` over the 16 popcounts) reads 0.99999964 — +the expected ceiling for equal-mass buckets, which is why D-NXG-9's rollover +timer must be read on FIXED boundaries after the distribution moves, never on +freshly re-quantiled ones (a re-quantiled histogram is flat by construction and +carries no signal). + +**Honest scope.** The column is audio, not a facet column — no facet fixture +exists in-tree (harvest 2026-09-05). The mechanism is value-type-agnostic; the +facet rerun is the next arm. `popcount_batch_u64` exists as a named T1 primitive +(D-NXG-2's audit half-closed: present, scalar `count_ones` sum, not yet the +POPCNT lane its comment promises). Mask bytes hot at this N: 189 184 B for 16 +bands — the plan's "16 × 8 KB" was at 65 536 rows; this column is 1.44× that. + ## 2026-09-05 — E-NXG-1 — the exposure meter is a nested mask set **Status:** PROPOSAL (design reading of measured code; no code changed). @@ -327,7 +368,7 @@ mechanisms disagreed by **58.2%**: | site | reading on `main` | |---|---| -| `.claude/board/AGENT_LOG.md:481` | 58.2% **disagreement** | +| `.claude/board/AGENT_LOG.md` under "2026-08-19 — oracle-boundary reassessment sweep" (the **58.2%** findings paragraph) | 58.2% **disagreement** | | `.claude/board/INTEGRATION_PLANS.md:910` | 58.2% **disagreement** | The source says the opposite. `MedCare-rs/crates/medcare-server/src/views/atlas.rs:465` diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 8f894fc28..d346c4744 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -5,10 +5,10 @@ every row is Queued until PROBE-NXG-HIST-1 / ROLL-1 / FLOOR-1 (plan §5) run. | D-id | deliverable | status | |---|---|---| -| D-NXG-1 | `NestedBands` sealed shape (T2), version-keyed, one owner | Queued | -| D-NXG-2 | T1 name audit: `popcount_words` present or minted; bucket = `mask_ternlog::<0x10>` by name | Queued | +| D-NXG-1 | `NestedBands` sealed shape (T2), version-keyed, one owner | **In progress** — structure measured by PROBE-NXG-HIST-1 (E-NXG-17, 2026-09-05); seal/ownership shape not yet built | +| D-NXG-2 | T1 name audit: `popcount_words` present or minted; bucket = `mask_ternlog::<0x10>` by name | **Half-closed 2026-09-05** — `ndarray::simd::popcount_batch_u64` exists (scalar); bucket-by-name proven bit-identical to `mask_andnot` (C3) | | D-NXG-3 | T1 `bisect_column_by_mask` partial-popcount bisection | Queued | -| D-NXG-4 | Prozentrang = bucket index; `shape × rank` computed from the slab | Queued | +| D-NXG-4 | Prozentrang = bucket index; `shape × rank` computed from the slab | **Measured 2026-09-05** — rank == partition point on 94 572/94 572 real rows (C2); not yet wired to any consumer | | D-NXG-5 | rollover: split on budget or entropy, merge on collapse, never rewrite | Queued | | D-NXG-6 | rank-derived reject floor; σ demoted to diagnostic | Queued | | D-NXG-7 | preheat by mask inheritance | Queued | diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 5422e92cf..a1bcec0cd 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -1,3 +1,34 @@ +## TD-SIGMA-PROBE-MASK-ALIGNMENT-AND-WORK-METRIC-1 (2026-09-05) — OPEN + +**`crates/lance-graph/examples/sigma_probe_masked_traverse.rs` (shipped in +#1176 as an inherited commit, `a11fd64b`, "measurement only") carries two +Codex P2 findings posted after the merge; neither is fixed here.** The example +was not authored in the #1176 session; it rode the designated branch. The +findings are recorded so the probe's numbers are not cited as evidence until +they are addressed. + +1. **Block mask is not aligned to the generated communities.** With the + defaults `n=96`, `blocks=8` each community spans 12 ids, but `mask_blocks` + is called with counts 1, 9 and 38, so `(0..count)` cuts through a community + in every row. The random-vs-block delta therefore does not isolate block + alignment, which is the one thing the row claims to measure. Fix shape: + pass the community size in and select whole communities, or choose + densities whose counts are multiples of 12. +2. **`sigma_result` measures output cardinality, not multiplication work.** + `GrBMatrix::mxm` visits every `(i, j)` and probes every entry of `a.row(i)`; + all successful products for one `(i, j)` collapse into at most one stored + nonzero. Two workloads with the same `nnz_masked / nnz_full` can have very + different masked-multiply costs, so the probe's stop/speedup conclusions + about mask pushdown are not supported by the ratio it prints. Fix shape: + count column probes / successful semiring ops, or time an inward-mask + implementation. + +**Why not fixed in #1178:** the example is not this session's work and #1176 +is merged; the honest move is a ledger row plus a pointer, not a drive-by +rewrite of someone else's measurement. Owner: the sigma-probe author. +Cross-ref: E-NXG-16 (independence assumption in popcount selectivity) — the +same "cardinality is not work" caveat, one probe over. + ## TD-SIGKER-CLIPPY-RED-ON-BASE-1 (2026-09-04) — RESOLVED 2026-09-04 **`crates/sigker` does not pass `cargo clippy --all-targets -- -D warnings`, and diff --git a/.claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md b/.claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md index 67beeb126..5a26bbf33 100644 --- a/.claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md +++ b/.claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md @@ -156,6 +156,10 @@ falsifier that would kill it. Ordered by how many rooms ahead it sits. 2. PROBE-NXG-HIST-1: build one `NestedBands` from a real facet column; assert bucket masks are disjoint, nested, and sum to N; assert rank = partition point (rooms 0, 16). + **GREEN 2026-09-05** on a real audio column (94 572 rows), not a facet + column — `crates/lance-graph-planner/examples/probe_nxg_hist_1.rs`, + E-NXG-17. Room 3 regraded there (midpoint σ over-reads 12 % on quantile + buckets). 3. PROBE-NXG-ROLL-1: rollover on a bimodal column; entropy-triggered vs budget-triggered split order (room 2, D-NXG-5). 4. PROBE-NXG-FLOOR-1: rank-derived floor vs `mu + 3σ` on the HHTL tiers @@ -166,7 +170,7 @@ falsifier that would kill it. Ordered by how many rooms ahead it sits. | D-id | deliverable | status | |---|---|---| -| D-NXG-1 | `NestedBands` sealed shape (T2), version-keyed, one owner | Queued | +| D-NXG-1 | `NestedBands` sealed shape (T2), version-keyed, one owner | In progress (structure measured, E-NXG-17) | | D-NXG-2 | T1 name audit: `popcount_words` present or minted; bucket = `mask_ternlog::<0x10>` by name | Queued | | D-NXG-3 | T1 `bisect_column_by_mask` partial-popcount bisection | Queued | | D-NXG-4 | Prozentrang = bucket index; `shape × rank` computed from the slab | Queued | diff --git a/crates/lance-graph-planner/examples/probe_nxg_hist_1.rs b/crates/lance-graph-planner/examples/probe_nxg_hist_1.rs new file mode 100644 index 000000000..f08ad8f2c --- /dev/null +++ b/crates/lance-graph-planner/examples/probe_nxg_hist_1.rs @@ -0,0 +1,264 @@ +//! **PROBE-NXG-HIST-1** — plan `.claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md` +//! §5 step 2, the first gate before any of D-NXG-4..12 leaves PROPOSAL. +//! +//! The claim under test (E-NXG-1): a Belichtungsmesser reading over N rows IS +//! a chain of nested row masks `M_0 ⊆ M_1 ⊆ … ⊆ M_{B-1}`; bucket `i` is +//! `M_i ∧ ¬M_{i-1}` = `mask_ternlog::` by NAME (E-NXG-2/3); and a +//! row's Prozentrang is the index of the innermost mask containing it — a +//! partition point over nested masks, never a sort (E-NXG-2). +//! +//! It changes no library code. It builds one `NestedBands` from a REAL value +//! column and asserts three pre-registered claims plus their anti-vacuity +//! guards. Everything else it prints is an OBSERVATION, not a claim. +//! +//! ```text +//! CARGO_PROFILE_DEV_DEBUG=0 cargo run -p lance-graph-planner --example probe_nxg_hist_1 [wav-path] +//! ``` +//! +//! # The column +//! +//! `data/tts-cascade/tts_real_output.wav` — 94 572 samples of real 16-bit +//! mono speech already checked into this repo. The probe reads `|sample|` as +//! an `i32` distance-like column. It is real (recorded audio, heavy-tailed, +//! not drawn from any generator this probe controls) — the truth-architect +//! rule against synthetic inputs is satisfied. It is NOT a facet column: the +//! plan's §1 gap "Prozentrang exists only in doctrine" is about the mechanism, +//! and the mechanism is value-type-agnostic. A facet-column rerun is the +//! obvious next arm once one is on disk (none is: the 2026-09-05 harvest found +//! no `.soa`/facet fixture in-tree). +//! +//! # Pre-registered claims +//! +//! - **C1 structure.** The B=16 band masks are nested (`M_i ∧ ¬M_{i+1}` is +//! empty for every i), the 16 bucket masks are pairwise disjoint, and the +//! bucket popcounts sum to exactly N. +//! - **C2 rank = partition point.** For EVERY row, the bucket index read off +//! the masks equals `boundaries.partition_point(|b| b < v)` computed from +//! the value directly. Anti-vacuity: at least 8 of 16 buckets are non-empty +//! and no bucket holds more than half the rows — a degenerate column would +//! make C2 true of a one-bucket histogram. +//! - **C3 bucket is one named immediate.** `mask_ternlog::(M_i, +//! M_{i-1}, M_{i-1})` is bit-identical to `mask_andnot(M_i, M_{i-1})` and to +//! the bucket the reference computed. Can-it-fire: the neighbouring +//! immediate `AND3` on the same operands must DIFFER for every i whose +//! lower band is non-empty — otherwise the immediate is decoration. +//! +//! Assertions run at the very END so every claim is measured before any can +//! abort the run (the entropy-census lesson: a probe that asserts inline hides +//! evidence). + +use lance_graph_contract::thought_atoms::normalized_entropy; +use ndarray::simd::ternlog::{AND3, AND_ANDNOT2}; +use ndarray::simd::{gt_i32_to_mask, mask_andnot, mask_ternlog, popcount_batch_u64}; + +const B: usize = 16; +const DEFAULT_WAV: &str = "data/tts-cascade/tts_real_output.wav"; +const WAV_HEADER: usize = 44; + +fn load_abs_samples(path: &str) -> Vec { + let bytes = std::fs::read(path).unwrap_or_else(|e| panic!("read {path}: {e}")); + assert!( + bytes.len() > WAV_HEADER && &bytes[0..4] == b"RIFF", + "{path}: not a RIFF file" + ); + assert_eq!(&bytes[8..12], b"WAVE", "{path}: not WAVE"); + let bits = u16::from_le_bytes([bytes[34], bytes[35]]); + let ch = u16::from_le_bytes([bytes[22], bytes[23]]); + assert_eq!((bits, ch), (16, 1), "{path}: expected 16-bit mono PCM"); + bytes[WAV_HEADER..] + .chunks_exact(2) + .map(|c| (i16::from_le_bytes([c[0], c[1]]) as i32).abs()) + .collect() +} + +fn words_for(n: usize) -> usize { + n.div_ceil(64) +} + +/// `M = rows with value <= boundary`, as `!gt` with the tail beyond `n` cleared. +fn le_mask(values: &[i32], boundary: i32) -> Vec { + let n = values.len(); + let mut m = vec![0u64; words_for(n)]; + gt_i32_to_mask(values, boundary, &mut m); + for w in m.iter_mut() { + *w = !*w; + } + if !n.is_multiple_of(64) { + let last = m.len() - 1; + m[last] &= (1u64 << (n % 64)) - 1; + } + m +} + +fn bit(m: &[u64], row: usize) -> bool { + (m[row / 64] >> (row % 64)) & 1 == 1 +} + +fn main() { + let path = std::env::args() + .nth(1) + .unwrap_or_else(|| DEFAULT_WAV.to_string()); + let values = load_abs_samples(&path); + let n = values.len(); + let words = words_for(n); + println!( + "PROBE-NXG-HIST-1\ncolumn: {path}\nrows N = {n}, mask words = {words}, bands B = {B}\n" + ); + + // ── boundaries: empirical quantiles (the ONE sort in this probe; it is the + // reference the mask walk must reproduce, never the mechanism) ─────────── + let mut sorted = values.clone(); + sorted.sort_unstable(); + let boundaries: Vec = (0..B) + .map(|i| sorted[((i + 1) * n / B).saturating_sub(1)]) + .collect(); + println!("boundaries (quantile): {boundaries:?}"); + + // ── the nested band masks M_0 ⊆ … ⊆ M_{B-1} ───────────────────────────── + let bands: Vec> = boundaries.iter().map(|&b| le_mask(&values, b)).collect(); + let band_pop: Vec = bands.iter().map(|m| popcount_batch_u64(m)).collect(); + + // ── buckets by the NAMED immediate, and two references ────────────────── + let mut buckets: Vec> = Vec::with_capacity(B); + let mut ref_andnot: Vec> = Vec::with_capacity(B); + let mut c3_named_eq_andnot = true; + let mut c3_and3_differs_where_it_can = true; + for i in 0..B { + let mut bucket = vec![0u64; words]; + let mut refb = vec![0u64; words]; + if i == 0 { + bucket.copy_from_slice(&bands[0]); + refb.copy_from_slice(&bands[0]); + } else { + mask_ternlog::(&bands[i], &bands[i - 1], &bands[i - 1], &mut bucket); + mask_andnot(&bands[i], &bands[i - 1], &mut refb); + let mut and3 = vec![0u64; words]; + mask_ternlog::(&bands[i], &bands[i - 1], &bands[i - 1], &mut and3); + if band_pop[i - 1] > 0 && and3 == bucket { + c3_and3_differs_where_it_can = false; + } + } + if bucket != refb { + c3_named_eq_andnot = false; + } + buckets.push(bucket); + ref_andnot.push(refb); + } + let bucket_pop: Vec = buckets.iter().map(|m| popcount_batch_u64(m)).collect(); + + // ── C1: nested, disjoint, sums to N ───────────────────────────────────── + let mut scratch = vec![0u64; words]; + let mut c1_nested = true; + for i in 0..B - 1 { + mask_andnot(&bands[i], &bands[i + 1], &mut scratch); + if popcount_batch_u64(&scratch) != 0 { + c1_nested = false; + } + } + let mut c1_disjoint = true; + for i in 0..B { + for j in i + 1..B { + // AND2 = a & b (c ignored); reuse c = a. + mask_ternlog::<{ ndarray::simd::ternlog::AND2 }>( + &buckets[i], + &buckets[j], + &buckets[i], + &mut scratch, + ); + if popcount_batch_u64(&scratch) != 0 { + c1_disjoint = false; + } + } + } + let c1_sum: u64 = bucket_pop.iter().sum(); + + // ── C2: rank read off the masks == partition point on the value ────────── + let mut c2_mismatches = 0usize; + for (row, &v) in values.iter().enumerate() { + let by_value = boundaries.partition_point(|&b| b < v).min(B - 1); + let by_mask = (0..B).find(|&i| bit(&buckets[i], row)); + if by_mask != Some(by_value) { + c2_mismatches += 1; + } + } + let nonempty = bucket_pop.iter().filter(|&&p| p > 0).count(); + let max_bucket = *bucket_pop.iter().max().unwrap_or(&0); + + // ── OBSERVATIONS (measured, not claims) ───────────────────────────────── + println!("\nOBSERVATIONS (measured, not claims)"); + println!(" band popcounts (cumulative): {band_pop:?}"); + println!(" bucket popcounts: {bucket_pop:?}"); + let weights: Vec = bucket_pop.iter().map(|&p| p as f32).collect(); + let h = normalized_entropy(&weights); + println!(" normalized Shannon entropy of the bucket histogram (D-NXG-9 preview): {h:?}"); + // σ recovered from the histogram (bucket midpoints weighted by popcount) + // vs σ computed directly — plan §3 room 3, observation only. + let mids: Vec = (0..B) + .map(|i| { + let lo = if i == 0 { + 0.0 + } else { + boundaries[i - 1] as f64 + }; + (lo + boundaries[i] as f64) / 2.0 + }) + .collect(); + let mean_h = mids + .iter() + .zip(&bucket_pop) + .map(|(m, &p)| m * p as f64) + .sum::() + / n as f64; + let var_h = mids + .iter() + .zip(&bucket_pop) + .map(|(m, &p)| (m - mean_h).powi(2) * p as f64) + .sum::() + / n as f64; + let mean_d = values.iter().map(|&v| v as f64).sum::() / n as f64; + let var_d = values + .iter() + .map(|&v| (v as f64 - mean_d).powi(2)) + .sum::() + / n as f64; + println!( + " sigma from histogram = {:.2}, sigma direct = {:.2}, ratio = {:.4} (room 3: unasserted)", + var_h.sqrt(), + var_d.sqrt(), + var_h.sqrt() / var_d.sqrt() + ); + println!( + " mask bytes hot: {} band masks x {} B = {} B", + B, + words * 8, + B * words * 8 + ); + + // ── verdicts, asserted LAST ────────────────────────────────────────────── + println!("\nCLAIMS"); + println!(" C1 nested={c1_nested} disjoint={c1_disjoint} sum={c1_sum} (N={n})"); + println!(" C2 rank mismatches = {c2_mismatches} / {n}; non-empty buckets = {nonempty}/{B}; max bucket = {max_bucket}"); + println!(" C3 named==andnot: {c3_named_eq_andnot}; AND3 differs where lower band non-empty: {c3_and3_differs_where_it_can}"); + + assert!(c1_nested, "C1: bands are not nested"); + assert!(c1_disjoint, "C1: buckets are not pairwise disjoint"); + assert_eq!(c1_sum, n as u64, "C1: bucket popcounts do not sum to N"); + assert!(nonempty >= 8, "C2 anti-vacuity: only {nonempty} of {B} buckets non-empty — the column is degenerate for this probe"); + assert!( + (max_bucket as usize) * 2 < n, + "C2 anti-vacuity: one bucket holds >= half the rows" + ); + assert_eq!( + c2_mismatches, 0, + "C2: mask-walk rank != partition-point rank" + ); + assert!( + c3_named_eq_andnot, + "C3: mask_ternlog:: != mask_andnot" + ); + assert!( + c3_and3_differs_where_it_can, + "C3 can-it-fire: AND3 gave the same mask as AND_ANDNOT2 — the immediate is inert" + ); + println!("\nC1 PASS · C2 PASS · C3 PASS."); +}