diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 796676f83..5bd02058c 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,59 @@ +## 2026-09-04 — E-A-SKETCH-THAT-MISSED-TWICE-WILL-MISS-A-THIRD-TIME-1 — the W1.5 lane-type sketches are 3-for-3 wrong + +**Status:** FINDING (read directly off `crates/sigker` source this session). +**Confidence:** High — every claim below is a file:line read, not an inference. + +**The measurement.** ndarray's consumer contract +(`.claude/knowledge/vertical-simd-consumer-contract.md`, W1.5) sketches three +SIMD primitives against `crates/sigker`. Each sketch names a lane type. All +three are wrong, the same way: + +| item | doc sketch | real consumer | shipped as | +|---|---|---|---| +| W1.5-#6 signature_pde | `&[F32x16]` | `f64` (`signature.rs`) | `F64x8` (PR #293) | +| W1.5-#7 randomized | "`F32x16` state" | `f64`/`Vec` | `F64x8` (PR #294) | +| W1.5-#8 lyndon-pack | "`I16x16` state" | `Vec` (`log_signature.rs:272`) | — not built | + +#8 is the worst of the three: `I16x16` is not merely imprecise, it is the wrong +element WIDTH and the wrong SIGNEDNESS FAMILY — there is no `i16` anywhere in +`log_signature.rs`. The crate is `f64` throughout. + +**The deeper finding — #8 is not a SIMD primitive at all.** The sketch names +"pack/unpack primitives", but no such operation exists in the consumer. The real +cost centres are `bracket_expansion` (`log_signature.rs:227-256`) — recursive +Lyndon-word splitting with a sparse outer-product merge plus `sort` and coalesce +over `(usize, f64)` pairs — and the scatter-add peel in +`project_onto_lyndon_basis` (`:368-386`), whose indices are data-dependent and +non-contiguous. Branchy recursion over variable-length sparse vectors is the +OPPOSITE of the dense, fixed-lane shape that made #6 (a 2D grid sweep) and #7 +(a `k×k` GEMV) good SIMD targets. The Lyndon basis is also generated on the fly +by Duval's algorithm at call time (`:164`, `:346`) with no memoization — so there +is no caller-owned table to consume the way #7 consumes projection buffers. + +**Where the real opportunity is, if any.** `log_signature.rs:100` names the +actual bottleneck itself: the depth-N Magnus expansion in `tensor_log`, i.e. +`tensor_multiply` in `signature.rs`, O(d^(2N)). That is a different primitive +from the one W1.5-#8 describes. + +**Bonus correction — the compression headline is misquoted.** The contract doc +says "7-13× compression, lossless". `log_signature.rs`'s own test +`compression_at_shallow_depth_is_far_below_the_headline` (`:744`) is a NAMED +FALSIFIER proving that figure is asymptotic (N≥8), not typical — at d=4,N=2 it +is ~2.1×. The crate's own header (`:41` area, and `lib.rs:29-31`) already flags +this conflation as an error it corrected. Any #8 spec citing "7-13×" +uncritically propagates a claim the source crate disowns. + +**Rule.** A doc sketch that has missed twice on the same axis is not a spec, it +is a hypothesis with a measured failure rate. Read the consumer source FIRST for +every remaining W1.5 item; treat the sketch as the thing to be checked, never as +the thing to implement. + +**Recommendation:** do NOT build W1.5-#8 as sketched. Before any ndarray +primitive signature is written, either (a) profile `log_signature_truncated` at +production depths to show `bracket_expansion`/peel is a measurable fraction of +wall time versus the Magnus series, or (b) rescope onto `tensor_multiply`. + +--- ## 2026-09-04 — E-A-GATE-INHERITS-THE-BLIND-SPOT-OF-WHOEVER-WROTE-IT-1 — both board gates shipped with the exact defect they gate against **Status:** FINDING (measured — three review findings, each verified against diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 9c62fe4ae..797036f7a 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -1,3 +1,54 @@ +## TD-SIGKER-CLIPPY-RED-ON-BASE-1 (2026-09-04) — OPEN + +**`crates/sigker` does not pass `cargo clippy --all-targets -- -D warnings`, and +did not before this session's changes.** Measured by stashing the working tree +and re-running against the base commit: the failure reproduces identically. + +Single site: `crates/sigker/src/signature.rs:133` — `for flat in 0..len` trips +clippy's needs_range_loop lint: the loop variable is used only to index `level`. + +(Anchored on the loop header, not on the lint name. The lint name is not text +that appears at the cited line, so citing it as the anchor is exactly the decay +`citation_decay.py` exists to catch — it flagged this entry's first draft.) + +```rust +for flat in 0..len { // clippy: needs_range_loop + let mut idx = flat; + ... + level[flat] = prod / factorial; +} +``` + +Proposed fix (3 lines, mechanical, deliberately NOT applied here to avoid +widening a wiring PR): + +```rust +for (flat, slot) in level.iter_mut().enumerate() { + let mut idx = flat; + ... + *slot = prod / factorial; +} +``` + +**Why it went unnoticed:** sigker is workspace-EXCLUDED, so `cargo clippy` at +the workspace root never reaches it — it needs +`--manifest-path crates/sigker/Cargo.toml`. Any excluded crate is outside the +default lint sweep; worth checking the other excluded crates (bgz17, +lance-graph-codec-research) for the same blind spot. + +**The same exclusion hid the TESTS too — closed in this PR.** `rust-test.yml` +enumerates workspace-excluded crates one scoped step at a time (deepnsm, +deepnsm-v2, supervisor, causal-edge, bgz-tensor, ogar, weather-poc, jc) and +sigker was simply never added. Its 62 tests — including BOTH W1.5 consumer +wirings, which are the crate's whole current purpose — had never run in CI. +A parity test against a scalar oracle that CI never executes is not a gate; +this is the same "blind gate" the file's own comments say the repo has closed +"one crate at a time". A tests-only step now arms it, on the causal-edge +precedent (that crate is also tests-only in CI precisely because its clippy is +red on arrival). The clippy half stays open, tracked by this entry. + +--- + ## TD-RELIABILITY-COPIES-OUTSIDE-JC-1 (2026-09-02) — OPEN **Two further copies of the Pearson / Spearman / Cronbach α / ICC battery live diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index 4a755dbb8..c9be8a552 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -198,6 +198,27 @@ jobs: # (deepnsm / supervisor / ogar above). One scoped step arms it. Gating. - name: Run bgz-tensor tests (workspace-excluded, ndarray sibling) run: cargo test --manifest-path crates/bgz-tensor/Cargo.toml --lib + # sigker: workspace-EXCLUDED path-signature codec (mandatory path dep on + # the ndarray sibling checked out above). None of the steps above reach + # it, so its 62 tests only ever ran on a developer machine — including + # the two W1.5 consumer wirings that are the whole point of the crate + # today: kernel.rs's signature_pde_sweep delegation (ndarray PR #293) and + # randomized.rs's randomized_signature_sweep delegation (ndarray PR #294). + # A parity test against a scalar oracle that CI never runs is not a gate. + # Same "blind gate" closed above for deepnsm / supervisor / causal-edge / + # bgz-tensor, one crate at a time. + # + # Verified locally before landing: 62 passed, 0 failed. + # + # Deliberately TESTS ONLY, on the causal-edge precedent above. A + # `clippy -D warnings` step would be red on arrival: the crate carries one + # pre-existing finding at signature.rs:133 (needs_range_loop), which + # reproduces identically on the base commit and is NOT in the + # randomized.rs this PR touches. Gating it here would fail this PR for a + # defect it did not introduce; it is recorded in TECH_DEBT + # TD-SIGKER-CLIPPY-RED-ON-BASE-1 instead, with its patch. + - name: Run sigker tests (workspace-excluded, ndarray sibling) + run: cargo test --manifest-path crates/sigker/Cargo.toml # lance-graph-callcenter UNDER `--features query` — the same blind gate as # supervisor above, but one level subtler: the crate was not merely # untested, it has `default = []`, so even a bare `cargo test` on it would diff --git a/crates/sigker/src/randomized.rs b/crates/sigker/src/randomized.rs index 788fe98a6..7700d02cd 100644 --- a/crates/sigker/src/randomized.rs +++ b/crates/sigker/src/randomized.rs @@ -46,6 +46,7 @@ //! signature is the trade: ~10000× more compute for guaranteed information //! preservation. +use ndarray::hpc::randomized_signature::randomized_signature_sweep; use std::f64::consts::PI; // ════════════════════════════════════════════════════════════════════════════ @@ -92,46 +93,35 @@ impl RandomizedSignatureBuilder { } /// Encode a path and return its randomized signature. + /// + /// # Implementation + /// + /// The recurrence `z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i)` is solved by + /// [`ndarray::hpc::randomized_signature::randomized_signature_sweep`] — + /// the SIMD GEMV sweep that replaced this function's original hand-rolled + /// row-major scalar loop (`TD-PILLAR11-SCIENTIFIC-LOOPS-BYPASS-NDARRAY-SIMD-1`: + /// numeric/computational code in the Ada stack is written against + /// `ndarray::simd::method()`, never a bespoke scalar loop). The buffer + /// layout is identical (`matrices[i*k*k + row*k + col]`, `biases` + /// concatenated per-axis) and the same `|Δx_i| < 1e-15` skip applies, so + /// the delegation carries sigker's numerics unchanged. + /// + /// # Panics + /// + /// Panics if `path` is empty or `path[0].len() != self.path_dim` — this + /// crate's original, stricter contract, unchanged by the delegation. + /// The delegated primitive additionally asserts (via `assert_eq!`, so it + /// fires in release builds too) that *every* point in `path` shares + /// `path[0]`'s dimension — a ragged-path guard this crate did not + /// previously enforce, so a ragged path that used to silently produce a + /// wrong (partially-truncated) signature now panics with "ragged path" + /// instead. This is a strict improvement, not a behavior-preserving + /// no-op — see the `ragged_path_now_panics` test below. pub fn encode(&self, path: &[Vec]) -> RandomizedSignature { assert!(!path.is_empty(), "path must have ≥1 point"); assert_eq!(path[0].len(), self.path_dim, "path point dim mismatch"); - let k = self.state_dim; - let d = self.path_dim; - let mut z = vec![0.0f64; k]; - - for window in path.windows(2) { - let delta_x: Vec = window[1] - .iter() - .zip(window[0].iter()) - .map(|(a, b)| a - b) - .collect(); - - // z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i) - let mut z_next = z.clone(); - let mut activated = vec![0.0f64; k]; - for i in 0..d { - let dx_i = delta_x[i]; - if dx_i.abs() < 1e-15 { - continue; - } - // activated = tanh(A_i · z + b_i) - let a_offset = i * k * k; - let b_offset = i * k; - for row in 0..k { - let mut sum = self.biases[b_offset + row]; - let row_off = a_offset + row * k; - for col in 0..k { - sum += self.matrices[row_off + col] * z[col]; - } - activated[row] = sum.tanh(); - } - for row in 0..k { - z_next[row] += activated[row] * dx_i; - } - } - z = z_next; - } + let z = randomized_signature_sweep(path, &self.matrices, &self.biases, self.state_dim); RandomizedSignature { path_dim: self.path_dim, @@ -263,4 +253,92 @@ mod tests { let s = b.encode(&path); assert_eq!(s.dim(), 128); } + + /// A small scalar oracle mirroring the ORIGINAL hand-rolled recurrence + /// this module used before delegating to + /// `ndarray::hpc::randomized_signature::randomized_signature_sweep`. + /// This test pins that `encode`'s delegated output still matches that + /// recurrence within a relative tolerance — it would fail if the + /// delegation silently changed the recurrence, the skip epsilon, the + /// activation, or the buffer-layout convention (row/col vs i/j swap). + /// The tolerance is NOT bit-equality: the SIMD GEMV reduces partial + /// sums and fuses products in a different order than this scalar loop, + /// so exact bit-equality is not guaranteed by construction. + fn scalar_reference( + path: &[Vec], + matrices: &[f64], + biases: &[f64], + state_dim: usize, + path_dim: usize, + ) -> Vec { + let k = state_dim; + let d = path_dim; + let mut z = vec![0.0f64; k]; + for window in path.windows(2) { + let delta_x: Vec = window[1] + .iter() + .zip(window[0].iter()) + .map(|(a, b)| a - b) + .collect(); + let mut z_next = z.clone(); + let mut activated = vec![0.0f64; k]; + for (i, &dx_i) in delta_x.iter().enumerate().take(d) { + if dx_i.abs() < 1e-15 { + continue; + } + let a_offset = i * k * k; + let b_offset = i * k; + for row in 0..k { + let mut sum = biases[b_offset + row]; + let row_off = a_offset + row * k; + for col in 0..k { + sum += matrices[row_off + col] * z[col]; + } + activated[row] = sum.tanh(); + } + for row in 0..k { + z_next[row] += activated[row] * dx_i; + } + } + z = z_next; + } + z + } + + #[test] + fn encode_matches_scalar_reference_within_tolerance() { + let b = RandomizedSignatureBuilder::new(3, 24, 0x5EED); + let path = vec![ + vec![0.0, 0.0, 0.0], + vec![1.0, -0.5, 0.25], + vec![0.7, 0.3, -1.1], + vec![2.0, 2.0, 2.0], + ]; + let s = b.encode(&path); + let expected = scalar_reference(&path, &b.matrices, &b.biases, b.state_dim, b.path_dim); + assert_eq!(s.state.len(), expected.len()); + for (got, want) in s.state.iter().zip(expected.iter()) { + let scale = want.abs().max(1.0); + assert!( + (got - want).abs() / scale < 1e-9, + "delegated encode diverged from scalar reference: got {got}, want {want}" + ); + } + } + + /// Establishes that `encode` now enforces the ragged-path guard the + /// delegated ndarray primitive carries (`assert_eq!`, so it fires in + /// release builds too) — a guard this crate's own asserts (which only + /// check `path[0]`) did not previously provide. Before the delegation + /// this exact input would NOT have panicked; it would have silently + /// read out-of-bounds-safe but semantically wrong deltas for the + /// shorter/longer point instead. This test discriminates the new + /// behavior from the old one, not merely restating the delegation. + #[test] + #[should_panic(expected = "ragged path")] + fn ragged_path_now_panics() { + let b = RandomizedSignatureBuilder::new(2, 8, 1); + let ragged = vec![vec![0.0, 0.0], vec![1.0, 1.0, 1.0]]; + let _ = b.encode(&ragged); + } }