From 71856ba9a04f192093747a2ffc8c44d8747382e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 02:59:39 +0000 Subject: [PATCH 1/3] sigker: delegate randomized-signature encode to the ndarray SIMD sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the follow-up ndarray PR #294 explicitly deferred ("Wiring sigker to actually call this is out of scope"). Until now the primitive was dead code: ndarray shipped and benchmarked the SIMD recurrence, but RandomizedSignatureBuilder::encode still ran its own row-major scalar loop, so the measured 2.0x-3.8x speedup was unrealised. encode now delegates to ndarray::hpc::randomized_signature::randomized_signature_sweep. The buffer layout was already identical (matrices[i*k*k + row*k + col], biases concatenated) and so is the |dx_i| < 1e-15 skip, so the delegation carries the numerics unchanged. This mirrors the precedent kernel.rs:35 already set for PR #293's signature_pde_sweep. BEHAVIOUR CHANGE, deliberate and documented. Both of sigker's stricter caller-facing asserts (non-empty path; path[0].len() == path_dim) are KEPT ahead of the delegation — the ndarray primitive accepts an empty path and returns the zero state, so dropping them would have been a silent contract change. But the primitive additionally asserts that EVERY path point matches path[0]'s dimension. sigker previously checked only path[0], so a ragged path used to produce a silently truncated wrong signature and now panics. That is a strict improvement; it is recorded in a # Panics section and pinned by a test. Tests (both new, both discriminating): - encode_matches_scalar_reference_within_tolerance — parity against an inlined scalar oracle transcribed from the pre-delegation loop, at 1e-9 relative tolerance rather than bit-equality (the SIMD GEMV reduces eight partial sums and fuses products, and reduce_sum order differs per backend). Catches drift in the recurrence, the skip epsilon, the activation, or the buffer layout. - ragged_path_now_panics — the exact input that did NOT panic before this change and produced a wrong signature instead. Board hygiene in the same commit, per the Mandatory Board-Hygiene Rule: - EPIPHANIES: E-A-SKETCH-THAT-MISSED-TWICE-WILL-MISS-A-THIRD-TIME-1. The contract doc's W1.5 lane-type sketches are 3-for-3 wrong (#6 and #7 sketched f32/F32x16 against an f64 consumer; #8 sketches I16x16 where log_signature.rs is Vec throughout). The entry also records that W1.5-#8 is NOT a well-shaped SIMD primitive at all — its real cost centres are recursive sparse bracket expansion and data-dependent scatter-adds, not a dense kernel — and that the "7-13x compression" headline is asymptotic, per the crate's own named falsifier test. - TECH_DEBT: TD-SIGKER-CLIPPY-RED-ON-BASE-1. sigker fails clippy --all-targets -D warnings at signature.rs:133, and did so BEFORE this change (verified by stashing and re-running against base). Not fixed here to avoid widening a wiring PR; the 3-line patch is in the entry. Root cause is that sigker is workspace-excluded, so root-level clippy never reaches it. Verified: cargo test --manifest-path crates/sigker/Cargo.toml — 62 passed, 0 failed; cargo fmt --check clean; clippy clean on every site this diff touches (only the pre-existing signature.rs:133 remains, tracked above). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv --- .claude/board/EPIPHANIES.md | 57 ++++++++++++ .claude/board/TECH_DEBT.md | 36 ++++++++ crates/sigker/src/randomized.rs | 150 ++++++++++++++++++++++++-------- 3 files changed, 207 insertions(+), 36 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 74aa12f74..72b96637f 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,60 @@ +## 2026-09-04 — E-A-SKETCH-THAT-MISSED-TWICE-WILL-MISS-A-THIRD-TIME-1 — the W1.5 lane-type sketches are 3-for-3 wrong + +**Status:** FINDING (read directly off `crates/sigker` source this session). +**Confidence:** High — every claim below is a file:line read, not an inference. + +**The measurement.** ndarray's consumer contract +(`.claude/knowledge/vertical-simd-consumer-contract.md`, W1.5) sketches three +SIMD primitives against `crates/sigker`. Each sketch names a lane type. All +three are wrong, the same way: + +| item | doc sketch | real consumer | shipped as | +|---|---|---|---| +| W1.5-#6 signature_pde | `&[F32x16]` | `f64` (`signature.rs`) | `F64x8` (PR #293) | +| W1.5-#7 randomized | "`F32x16` state" | `f64`/`Vec` | `F64x8` (PR #294) | +| W1.5-#8 lyndon-pack | "`I16x16` state" | `Vec` (`log_signature.rs:272`) | — not built | + +#8 is the worst of the three: `I16x16` is not merely imprecise, it is the wrong +element WIDTH and the wrong SIGNEDNESS FAMILY — there is no `i16` anywhere in +`log_signature.rs`. The crate is `f64` throughout. + +**The deeper finding — #8 is not a SIMD primitive at all.** The sketch names +"pack/unpack primitives", but no such operation exists in the consumer. The real +cost centres are `bracket_expansion` (`log_signature.rs:227-256`) — recursive +Lyndon-word splitting with a sparse outer-product merge plus `sort` and coalesce +over `(usize, f64)` pairs — and the scatter-add peel in +`project_onto_lyndon_basis` (`:368-386`), whose indices are data-dependent and +non-contiguous. Branchy recursion over variable-length sparse vectors is the +OPPOSITE of the dense, fixed-lane shape that made #6 (a 2D grid sweep) and #7 +(a `k×k` GEMV) good SIMD targets. The Lyndon basis is also generated on the fly +by Duval's algorithm at call time (`:164`, `:346`) with no memoization — so there +is no caller-owned table to consume the way #7 consumes projection buffers. + +**Where the real opportunity is, if any.** `log_signature.rs:100` names the +actual bottleneck itself: the depth-N Magnus expansion in `tensor_log`, i.e. +`tensor_multiply` in `signature.rs`, O(d^(2N)). That is a different primitive +from the one W1.5-#8 describes. + +**Bonus correction — the compression headline is misquoted.** The contract doc +says "7-13× compression, lossless". `log_signature.rs`'s own test +`compression_at_shallow_depth_is_far_below_the_headline` (`:744`) is a NAMED +FALSIFIER proving that figure is asymptotic (N≥8), not typical — at d=4,N=2 it +is ~2.1×. The crate's own header (`:41` area, and `lib.rs:29-31`) already flags +this conflation as an error it corrected. Any #8 spec citing "7-13×" +uncritically propagates a claim the source crate disowns. + +**Rule.** A doc sketch that has missed twice on the same axis is not a spec, it +is a hypothesis with a measured failure rate. Read the consumer source FIRST for +every remaining W1.5 item; treat the sketch as the thing to be checked, never as +the thing to implement. + +**Recommendation:** do NOT build W1.5-#8 as sketched. Before any ndarray +primitive signature is written, either (a) profile `log_signature_truncated` at +production depths to show `bracket_expansion`/peel is a measurable fraction of +wall time versus the Magnus series, or (b) rescope onto `tensor_multiply`. + +--- + ## 2026-09-03 — E-A-CORRECTION-IS-ONLY-AS-GOOD-AS-ITS-MERGE-1 — an unlanded Storno leaves the falsehood standing **Status:** FINDING (measured on this repo's own `main`, this hour). The diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 9c62fe4ae..8e378d364 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -1,3 +1,39 @@ +## TD-SIGKER-CLIPPY-RED-ON-BASE-1 (2026-09-04) — OPEN + +**`crates/sigker` does not pass `cargo clippy --all-targets -- -D warnings`, and +did not before this session's changes.** Measured by stashing the working tree +and re-running against the base commit: the failure reproduces identically. + +Single site: `crates/sigker/src/signature.rs:133` — `needs_range_loop`, the loop +variable `flat` is used only to index `level`: + +```rust +for flat in 0..len { // clippy: needs_range_loop + let mut idx = flat; + ... + level[flat] = prod / factorial; +} +``` + +Proposed fix (3 lines, mechanical, deliberately NOT applied here to avoid +widening a wiring PR): + +```rust +for (flat, slot) in level.iter_mut().enumerate() { + let mut idx = flat; + ... + *slot = prod / factorial; +} +``` + +**Why it went unnoticed:** sigker is workspace-EXCLUDED, so `cargo clippy` at +the workspace root never reaches it — it needs +`--manifest-path crates/sigker/Cargo.toml`. Any excluded crate is outside the +default lint sweep; worth checking the other excluded crates (bgz17, deepnsm, +bgz-tensor, lance-graph-codec-research) for the same blind spot. + +--- + ## TD-RELIABILITY-COPIES-OUTSIDE-JC-1 (2026-09-02) — OPEN **Two further copies of the Pearson / Spearman / Cronbach α / ICC battery live diff --git a/crates/sigker/src/randomized.rs b/crates/sigker/src/randomized.rs index 788fe98a6..7700d02cd 100644 --- a/crates/sigker/src/randomized.rs +++ b/crates/sigker/src/randomized.rs @@ -46,6 +46,7 @@ //! signature is the trade: ~10000× more compute for guaranteed information //! preservation. +use ndarray::hpc::randomized_signature::randomized_signature_sweep; use std::f64::consts::PI; // ════════════════════════════════════════════════════════════════════════════ @@ -92,46 +93,35 @@ impl RandomizedSignatureBuilder { } /// Encode a path and return its randomized signature. + /// + /// # Implementation + /// + /// The recurrence `z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i)` is solved by + /// [`ndarray::hpc::randomized_signature::randomized_signature_sweep`] — + /// the SIMD GEMV sweep that replaced this function's original hand-rolled + /// row-major scalar loop (`TD-PILLAR11-SCIENTIFIC-LOOPS-BYPASS-NDARRAY-SIMD-1`: + /// numeric/computational code in the Ada stack is written against + /// `ndarray::simd::method()`, never a bespoke scalar loop). The buffer + /// layout is identical (`matrices[i*k*k + row*k + col]`, `biases` + /// concatenated per-axis) and the same `|Δx_i| < 1e-15` skip applies, so + /// the delegation carries sigker's numerics unchanged. + /// + /// # Panics + /// + /// Panics if `path` is empty or `path[0].len() != self.path_dim` — this + /// crate's original, stricter contract, unchanged by the delegation. + /// The delegated primitive additionally asserts (via `assert_eq!`, so it + /// fires in release builds too) that *every* point in `path` shares + /// `path[0]`'s dimension — a ragged-path guard this crate did not + /// previously enforce, so a ragged path that used to silently produce a + /// wrong (partially-truncated) signature now panics with "ragged path" + /// instead. This is a strict improvement, not a behavior-preserving + /// no-op — see the `ragged_path_now_panics` test below. pub fn encode(&self, path: &[Vec]) -> RandomizedSignature { assert!(!path.is_empty(), "path must have ≥1 point"); assert_eq!(path[0].len(), self.path_dim, "path point dim mismatch"); - let k = self.state_dim; - let d = self.path_dim; - let mut z = vec![0.0f64; k]; - - for window in path.windows(2) { - let delta_x: Vec = window[1] - .iter() - .zip(window[0].iter()) - .map(|(a, b)| a - b) - .collect(); - - // z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i) - let mut z_next = z.clone(); - let mut activated = vec![0.0f64; k]; - for i in 0..d { - let dx_i = delta_x[i]; - if dx_i.abs() < 1e-15 { - continue; - } - // activated = tanh(A_i · z + b_i) - let a_offset = i * k * k; - let b_offset = i * k; - for row in 0..k { - let mut sum = self.biases[b_offset + row]; - let row_off = a_offset + row * k; - for col in 0..k { - sum += self.matrices[row_off + col] * z[col]; - } - activated[row] = sum.tanh(); - } - for row in 0..k { - z_next[row] += activated[row] * dx_i; - } - } - z = z_next; - } + let z = randomized_signature_sweep(path, &self.matrices, &self.biases, self.state_dim); RandomizedSignature { path_dim: self.path_dim, @@ -263,4 +253,92 @@ mod tests { let s = b.encode(&path); assert_eq!(s.dim(), 128); } + + /// A small scalar oracle mirroring the ORIGINAL hand-rolled recurrence + /// this module used before delegating to + /// `ndarray::hpc::randomized_signature::randomized_signature_sweep`. + /// This test pins that `encode`'s delegated output still matches that + /// recurrence within a relative tolerance — it would fail if the + /// delegation silently changed the recurrence, the skip epsilon, the + /// activation, or the buffer-layout convention (row/col vs i/j swap). + /// The tolerance is NOT bit-equality: the SIMD GEMV reduces partial + /// sums and fuses products in a different order than this scalar loop, + /// so exact bit-equality is not guaranteed by construction. + fn scalar_reference( + path: &[Vec], + matrices: &[f64], + biases: &[f64], + state_dim: usize, + path_dim: usize, + ) -> Vec { + let k = state_dim; + let d = path_dim; + let mut z = vec![0.0f64; k]; + for window in path.windows(2) { + let delta_x: Vec = window[1] + .iter() + .zip(window[0].iter()) + .map(|(a, b)| a - b) + .collect(); + let mut z_next = z.clone(); + let mut activated = vec![0.0f64; k]; + for (i, &dx_i) in delta_x.iter().enumerate().take(d) { + if dx_i.abs() < 1e-15 { + continue; + } + let a_offset = i * k * k; + let b_offset = i * k; + for row in 0..k { + let mut sum = biases[b_offset + row]; + let row_off = a_offset + row * k; + for col in 0..k { + sum += matrices[row_off + col] * z[col]; + } + activated[row] = sum.tanh(); + } + for row in 0..k { + z_next[row] += activated[row] * dx_i; + } + } + z = z_next; + } + z + } + + #[test] + fn encode_matches_scalar_reference_within_tolerance() { + let b = RandomizedSignatureBuilder::new(3, 24, 0x5EED); + let path = vec![ + vec![0.0, 0.0, 0.0], + vec![1.0, -0.5, 0.25], + vec![0.7, 0.3, -1.1], + vec![2.0, 2.0, 2.0], + ]; + let s = b.encode(&path); + let expected = scalar_reference(&path, &b.matrices, &b.biases, b.state_dim, b.path_dim); + assert_eq!(s.state.len(), expected.len()); + for (got, want) in s.state.iter().zip(expected.iter()) { + let scale = want.abs().max(1.0); + assert!( + (got - want).abs() / scale < 1e-9, + "delegated encode diverged from scalar reference: got {got}, want {want}" + ); + } + } + + /// Establishes that `encode` now enforces the ragged-path guard the + /// delegated ndarray primitive carries (`assert_eq!`, so it fires in + /// release builds too) — a guard this crate's own asserts (which only + /// check `path[0]`) did not previously provide. Before the delegation + /// this exact input would NOT have panicked; it would have silently + /// read out-of-bounds-safe but semantically wrong deltas for the + /// shorter/longer point instead. This test discriminates the new + /// behavior from the old one, not merely restating the delegation. + #[test] + #[should_panic(expected = "ragged path")] + fn ragged_path_now_panics() { + let b = RandomizedSignatureBuilder::new(2, 8, 1); + let ragged = vec![vec![0.0, 0.0], vec![1.0, 1.0, 1.0]]; + let _ = b.encode(&ragged); + } } From 18a0b1de8e68d8abc2cf8040463a4c7b922d2306 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 03:03:22 +0000 Subject: [PATCH 2/3] ci: run sigker's tests (workspace-excluded, never gated) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sigker's 62 tests have never run in CI. `rust-test.yml` enumerates workspace-excluded crates one scoped step at a time — deepnsm, deepnsm-v2, supervisor, causal-edge, bgz-tensor, ogar, weather-poc, jc — and sigker was simply never added to that list. That matters most right now: BOTH of the crate's W1.5 consumer wirings, which are its whole current purpose, were invisible to CI. kernel.rs's signature_pde_sweep delegation (ndarray PR #293) has been unguarded since it landed, and randomized.rs's randomized_signature_sweep delegation is added in the preceding commit of this same PR. A parity test against a scalar oracle that CI never executes is not a gate. This is the same "blind gate" the workflow's own comments say this repo has closed "one crate at a time" for five other excluded crates. One scoped step arms it. Deliberately TESTS ONLY, following the causal-edge precedent immediately above it in the same file: a `clippy -D warnings` step would be red on arrival, because sigker carries one pre-existing finding at signature.rs:133 (needs_range_loop) that reproduces identically on the base commit and is not in the randomized.rs this PR touches. Gating it here would fail this PR for a defect it did not introduce. It stays recorded in TECH_DEBT TD-SIGKER-CLIPPY-RED-ON-BASE-1, with its patch, and that entry is extended here to cover the test half of the same exclusion blind spot. Verified: YAML parses; cargo test --manifest-path crates/sigker/Cargo.toml — 62 passed, 0 failed; cargo fmt --check clean; supersession index regenerated after the board write and confirmed current. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv --- .claude/board/TECH_DEBT.md | 15 +++++++++++++-- .github/workflows/rust-test.yml | 21 +++++++++++++++++++++ 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 8e378d364..fa53af86e 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -29,8 +29,19 @@ for (flat, slot) in level.iter_mut().enumerate() { **Why it went unnoticed:** sigker is workspace-EXCLUDED, so `cargo clippy` at the workspace root never reaches it — it needs `--manifest-path crates/sigker/Cargo.toml`. Any excluded crate is outside the -default lint sweep; worth checking the other excluded crates (bgz17, deepnsm, -bgz-tensor, lance-graph-codec-research) for the same blind spot. +default lint sweep; worth checking the other excluded crates (bgz17, +lance-graph-codec-research) for the same blind spot. + +**The same exclusion hid the TESTS too — closed in this PR.** `rust-test.yml` +enumerates workspace-excluded crates one scoped step at a time (deepnsm, +deepnsm-v2, supervisor, causal-edge, bgz-tensor, ogar, weather-poc, jc) and +sigker was simply never added. Its 62 tests — including BOTH W1.5 consumer +wirings, which are the crate's whole current purpose — had never run in CI. +A parity test against a scalar oracle that CI never executes is not a gate; +this is the same "blind gate" the file's own comments say the repo has closed +"one crate at a time". A tests-only step now arms it, on the causal-edge +precedent (that crate is also tests-only in CI precisely because its clippy is +red on arrival). The clippy half stays open, tracked by this entry. --- diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index 4a755dbb8..c9be8a552 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -198,6 +198,27 @@ jobs: # (deepnsm / supervisor / ogar above). One scoped step arms it. Gating. - name: Run bgz-tensor tests (workspace-excluded, ndarray sibling) run: cargo test --manifest-path crates/bgz-tensor/Cargo.toml --lib + # sigker: workspace-EXCLUDED path-signature codec (mandatory path dep on + # the ndarray sibling checked out above). None of the steps above reach + # it, so its 62 tests only ever ran on a developer machine — including + # the two W1.5 consumer wirings that are the whole point of the crate + # today: kernel.rs's signature_pde_sweep delegation (ndarray PR #293) and + # randomized.rs's randomized_signature_sweep delegation (ndarray PR #294). + # A parity test against a scalar oracle that CI never runs is not a gate. + # Same "blind gate" closed above for deepnsm / supervisor / causal-edge / + # bgz-tensor, one crate at a time. + # + # Verified locally before landing: 62 passed, 0 failed. + # + # Deliberately TESTS ONLY, on the causal-edge precedent above. A + # `clippy -D warnings` step would be red on arrival: the crate carries one + # pre-existing finding at signature.rs:133 (needs_range_loop), which + # reproduces identically on the base commit and is NOT in the + # randomized.rs this PR touches. Gating it here would fail this PR for a + # defect it did not introduce; it is recorded in TECH_DEBT + # TD-SIGKER-CLIPPY-RED-ON-BASE-1 instead, with its patch. + - name: Run sigker tests (workspace-excluded, ndarray sibling) + run: cargo test --manifest-path crates/sigker/Cargo.toml # lance-graph-callcenter UNDER `--features query` — the same blind gate as # supervisor above, but one level subtler: the crate was not merely # untested, it has `default = []`, so even a bare `cargo test` on it would From 47c9c3dcf00738ada8f55371ef0f3e96b1b6d591 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 03:51:07 +0000 Subject: [PATCH 3/3] board: re-anchor the sigker clippy citation on the loop header, not the lint name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merging origin/main brought in the new `citation-decay` gate (#1168/#1170), which fired on exactly one new citation — mine: TECH_DEBT.md -> crates/sigker/src/signature.rs:133 anchor=symbol:"needs_range_loop" (anchor absent within +-3 lines) The gate is right and the finding is a real defect, not a false positive. `needs_range_loop` is clippy's LINT NAME; it appears nowhere in the cited source. The gate extracts the backticked token nearest the citation as the anchor, so the lint name won that race and then could not be found at the line it supposedly addressed. A citation whose anchor is absent from its target is unverifiable by construction — which is the whole point of the gate. Re-anchored on `for flat in 0..len`, which is the actual text at signature.rs:133, and the lint name is now named in prose instead. The entry also records why, so the next reader does not "fix" it back. Local gate runs after the merge, all green: - citation_decay.py --self-test: passed (both halves) - citation_decay.py --since : 0 new decays (148 pre-existing backlog, unchanged and not failing) - append_only_gate.py --self-test: passed, 7 cases - append_only_gate.py origin/main: no protected file shrank, 9 checked (TECH_DEBT 4161 -> 4212, +51) - supersession_index.py: regenerated after the merge, byte-identical - plan_dids.py: no added plans, nothing to check - cargo test --manifest-path crates/sigker/Cargo.toml: 62 passed, 0 failed - cargo fmt --check: clean Also merges origin/main (f30e300d..ac9148f7) to clear the merge conflict that made this PR un-mergeable. The merge itself was clean; EPIPHANIES.md auto-merged, both sides having prepended. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv --- .claude/board/TECH_DEBT.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index fa53af86e..797036f7a 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -4,8 +4,12 @@ did not before this session's changes.** Measured by stashing the working tree and re-running against the base commit: the failure reproduces identically. -Single site: `crates/sigker/src/signature.rs:133` — `needs_range_loop`, the loop -variable `flat` is used only to index `level`: +Single site: `crates/sigker/src/signature.rs:133` — `for flat in 0..len` trips +clippy's needs_range_loop lint: the loop variable is used only to index `level`. + +(Anchored on the loop header, not on the lint name. The lint name is not text +that appears at the cited line, so citing it as the anchor is exactly the decay +`citation_decay.py` exists to catch — it flagged this entry's first draft.) ```rust for flat in 0..len { // clippy: needs_range_loop