sigker: delegate randomized-signature encode to the ndarray SIMD sweep

.claude/board/EPIPHANIES.md

@@ -1,3 +1,59 @@
+## 2026-09-04 — E-A-SKETCH-THAT-MISSED-TWICE-WILL-MISS-A-THIRD-TIME-1 — the W1.5 lane-type sketches are 3-for-3 wrong
+
+**Status:** FINDING (read directly off `crates/sigker` source this session).
+**Confidence:** High — every claim below is a file:line read, not an inference.
+
+**The measurement.** ndarray's consumer contract
+(`.claude/knowledge/vertical-simd-consumer-contract.md`, W1.5) sketches three
+SIMD primitives against `crates/sigker`. Each sketch names a lane type. All
+three are wrong, the same way:
+
+| item | doc sketch | real consumer | shipped as |
+|---|---|---|---|
+| W1.5-#6 signature_pde | `&[F32x16]` | `f64` (`signature.rs`) | `F64x8` (PR #293) |
+| W1.5-#7 randomized | "`F32x16` state" | `f64`/`Vec<f64>` | `F64x8` (PR #294) |
+| W1.5-#8 lyndon-pack | "`I16x16` state" | `Vec<f64>` (`log_signature.rs:272`) | — not built |
+
+#8 is the worst of the three: `I16x16` is not merely imprecise, it is the wrong
+element WIDTH and the wrong SIGNEDNESS FAMILY — there is no `i16` anywhere in
+`log_signature.rs`. The crate is `f64` throughout.
+
+**The deeper finding — #8 is not a SIMD primitive at all.** The sketch names
+"pack/unpack primitives", but no such operation exists in the consumer. The real
+cost centres are `bracket_expansion` (`log_signature.rs:227-256`) — recursive
+Lyndon-word splitting with a sparse outer-product merge plus `sort` and coalesce
+over `(usize, f64)` pairs — and the scatter-add peel in
+`project_onto_lyndon_basis` (`:368-386`), whose indices are data-dependent and
+non-contiguous. Branchy recursion over variable-length sparse vectors is the
+OPPOSITE of the dense, fixed-lane shape that made #6 (a 2D grid sweep) and #7
+(a `k×k` GEMV) good SIMD targets. The Lyndon basis is also generated on the fly
+by Duval's algorithm at call time (`:164`, `:346`) with no memoization — so there
+is no caller-owned table to consume the way #7 consumes projection buffers.
+
+**Where the real opportunity is, if any.** `log_signature.rs:100` names the
+actual bottleneck itself: the depth-N Magnus expansion in `tensor_log`, i.e.
+`tensor_multiply` in `signature.rs`, O(d^(2N)). That is a different primitive
+from the one W1.5-#8 describes.
+
+**Bonus correction — the compression headline is misquoted.** The contract doc
+says "7-13× compression, lossless". `log_signature.rs`'s own test
+`compression_at_shallow_depth_is_far_below_the_headline` (`:744`) is a NAMED
+FALSIFIER proving that figure is asymptotic (N≥8), not typical — at d=4,N=2 it
+is ~2.1×. The crate's own header (`:41` area, and `lib.rs:29-31`) already flags
+this conflation as an error it corrected. Any #8 spec citing "7-13×"
+uncritically propagates a claim the source crate disowns.
+
+**Rule.** A doc sketch that has missed twice on the same axis is not a spec, it
+is a hypothesis with a measured failure rate. Read the consumer source FIRST for
+every remaining W1.5 item; treat the sketch as the thing to be checked, never as
+the thing to implement.
+
+**Recommendation:** do NOT build W1.5-#8 as sketched. Before any ndarray
+primitive signature is written, either (a) profile `log_signature_truncated` at
+production depths to show `bracket_expansion`/peel is a measurable fraction of
+wall time versus the Magnus series, or (b) rescope onto `tensor_multiply`.
+
+---
 ## 2026-09-04 — E-A-GATE-INHERITS-THE-BLIND-SPOT-OF-WHOEVER-WROTE-IT-1 — both board gates shipped with the exact defect they gate against
 
 **Status:** FINDING (measured — three review findings, each verified against

.claude/board/TECH_DEBT.md

@@ -1,3 +1,54 @@
+## TD-SIGKER-CLIPPY-RED-ON-BASE-1 (2026-09-04) — OPEN
+
+**`crates/sigker` does not pass `cargo clippy --all-targets -- -D warnings`, and
+did not before this session's changes.** Measured by stashing the working tree
+and re-running against the base commit: the failure reproduces identically.
+
+Single site: `crates/sigker/src/signature.rs:133` — `for flat in 0..len` trips
+clippy's needs_range_loop lint: the loop variable is used only to index `level`.
+
+(Anchored on the loop header, not on the lint name. The lint name is not text
+that appears at the cited line, so citing it as the anchor is exactly the decay
+`citation_decay.py` exists to catch — it flagged this entry's first draft.)
+
+```rust
+for flat in 0..len {          // clippy: needs_range_loop
+    let mut idx = flat;
+    ...
+    level[flat] = prod / factorial;
+}
+```
+
+Proposed fix (3 lines, mechanical, deliberately NOT applied here to avoid
+widening a wiring PR):
+
+```rust
+for (flat, slot) in level.iter_mut().enumerate() {
+    let mut idx = flat;
+    ...
+    *slot = prod / factorial;
+}
+```
+
+**Why it went unnoticed:** sigker is workspace-EXCLUDED, so `cargo clippy` at
+the workspace root never reaches it — it needs
+`--manifest-path crates/sigker/Cargo.toml`. Any excluded crate is outside the
+default lint sweep; worth checking the other excluded crates (bgz17,
+lance-graph-codec-research) for the same blind spot.
+
+**The same exclusion hid the TESTS too — closed in this PR.** `rust-test.yml`
+enumerates workspace-excluded crates one scoped step at a time (deepnsm,
+deepnsm-v2, supervisor, causal-edge, bgz-tensor, ogar, weather-poc, jc) and
+sigker was simply never added. Its 62 tests — including BOTH W1.5 consumer
+wirings, which are the crate's whole current purpose — had never run in CI.
+A parity test against a scalar oracle that CI never executes is not a gate;
+this is the same "blind gate" the file's own comments say the repo has closed
+"one crate at a time". A tests-only step now arms it, on the causal-edge
+precedent (that crate is also tests-only in CI precisely because its clippy is
+red on arrival). The clippy half stays open, tracked by this entry.
+
+---
+
 ## TD-RELIABILITY-COPIES-OUTSIDE-JC-1 (2026-09-02) — OPEN
 
 **Two further copies of the Pearson / Spearman / Cronbach α / ICC battery live

.github/workflows/rust-test.yml

@@ -198,6 +198,27 @@ jobs:
       # (deepnsm / supervisor / ogar above). One scoped step arms it. Gating.
       - name: Run bgz-tensor tests (workspace-excluded, ndarray sibling)
         run: cargo test --manifest-path crates/bgz-tensor/Cargo.toml --lib
+      # sigker: workspace-EXCLUDED path-signature codec (mandatory path dep on
+      # the ndarray sibling checked out above). None of the steps above reach
+      # it, so its 62 tests only ever ran on a developer machine — including
+      # the two W1.5 consumer wirings that are the whole point of the crate
+      # today: kernel.rs's signature_pde_sweep delegation (ndarray PR #293) and
+      # randomized.rs's randomized_signature_sweep delegation (ndarray PR #294).
+      # A parity test against a scalar oracle that CI never runs is not a gate.
+      # Same "blind gate" closed above for deepnsm / supervisor / causal-edge /
+      # bgz-tensor, one crate at a time.
+      #
+      # Verified locally before landing: 62 passed, 0 failed.
+      #
+      # Deliberately TESTS ONLY, on the causal-edge precedent above. A
+      # `clippy -D warnings` step would be red on arrival: the crate carries one
+      # pre-existing finding at signature.rs:133 (needs_range_loop), which
+      # reproduces identically on the base commit and is NOT in the
+      # randomized.rs this PR touches. Gating it here would fail this PR for a
+      # defect it did not introduce; it is recorded in TECH_DEBT
+      # TD-SIGKER-CLIPPY-RED-ON-BASE-1 instead, with its patch.
+      - name: Run sigker tests (workspace-excluded, ndarray sibling)
+        run: cargo test --manifest-path crates/sigker/Cargo.toml
       # lance-graph-callcenter UNDER `--features query` — the same blind gate as
       # supervisor above, but one level subtler: the crate was not merely
       # untested, it has `default = []`, so even a bare `cargo test` on it would

crates/sigker/src/randomized.rs

@@ -46,6 +46,7 @@
 //! signature is the trade: ~10000× more compute for guaranteed information
 //! preservation.
 
+use ndarray::hpc::randomized_signature::randomized_signature_sweep;
 use std::f64::consts::PI;
 
 // ════════════════════════════════════════════════════════════════════════════
@@ -92,46 +93,35 @@ impl RandomizedSignatureBuilder {
     }
 
     /// Encode a path and return its randomized signature.
+    ///
+    /// # Implementation
+    ///
+    /// The recurrence `z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i)` is solved by
+    /// [`ndarray::hpc::randomized_signature::randomized_signature_sweep`] —
+    /// the SIMD GEMV sweep that replaced this function's original hand-rolled
+    /// row-major scalar loop (`TD-PILLAR11-SCIENTIFIC-LOOPS-BYPASS-NDARRAY-SIMD-1`:
+    /// numeric/computational code in the Ada stack is written against
+    /// `ndarray::simd::method()`, never a bespoke scalar loop). The buffer
+    /// layout is identical (`matrices[i*k*k + row*k + col]`, `biases`
+    /// concatenated per-axis) and the same `|Δx_i| < 1e-15` skip applies, so
+    /// the delegation carries sigker's numerics unchanged.
+    ///
+    /// # Panics
+    ///
+    /// Panics if `path` is empty or `path[0].len() != self.path_dim` — this
+    /// crate's original, stricter contract, unchanged by the delegation.
+    /// The delegated primitive additionally asserts (via `assert_eq!`, so it
+    /// fires in release builds too) that *every* point in `path` shares
+    /// `path[0]`'s dimension — a ragged-path guard this crate did not
+    /// previously enforce, so a ragged path that used to silently produce a
+    /// wrong (partially-truncated) signature now panics with "ragged path"
+    /// instead. This is a strict improvement, not a behavior-preserving
+    /// no-op — see the `ragged_path_now_panics` test below.
     pub fn encode(&self, path: &[Vec<f64>]) -> RandomizedSignature {
         assert!(!path.is_empty(), "path must have ≥1 point");
         assert_eq!(path[0].len(), self.path_dim, "path point dim mismatch");
 
-        let k = self.state_dim;
-        let d = self.path_dim;
-        let mut z = vec![0.0f64; k];
-
-        for window in path.windows(2) {
-            let delta_x: Vec<f64> = window[1]
-                .iter()
-                .zip(window[0].iter())
-                .map(|(a, b)| a - b)
-                .collect();
-
-            // z ← z + Σ_i tanh(A_i · z + b_i) · Δx^(i)
-            let mut z_next = z.clone();
-            let mut activated = vec![0.0f64; k];
-            for i in 0..d {
-                let dx_i = delta_x[i];
-                if dx_i.abs() < 1e-15 {
-                    continue;
-                }
-                // activated = tanh(A_i · z + b_i)
-                let a_offset = i * k * k;
-                let b_offset = i * k;
-                for row in 0..k {
-                    let mut sum = self.biases[b_offset + row];
-                    let row_off = a_offset + row * k;
-                    for col in 0..k {
-                        sum += self.matrices[row_off + col] * z[col];
-                    }
-                    activated[row] = sum.tanh();
-                }
-                for row in 0..k {
-                    z_next[row] += activated[row] * dx_i;
-                }
-            }
-            z = z_next;
-        }
+        let z = randomized_signature_sweep(path, &self.matrices, &self.biases, self.state_dim);
 
         RandomizedSignature {
             path_dim: self.path_dim,
@@ -263,4 +253,92 @@ mod tests {
         let s = b.encode(&path);
         assert_eq!(s.dim(), 128);
     }
+
+    /// A small scalar oracle mirroring the ORIGINAL hand-rolled recurrence
+    /// this module used before delegating to
+    /// `ndarray::hpc::randomized_signature::randomized_signature_sweep`.
+    /// This test pins that `encode`'s delegated output still matches that
+    /// recurrence within a relative tolerance — it would fail if the
+    /// delegation silently changed the recurrence, the skip epsilon, the
+    /// activation, or the buffer-layout convention (row/col vs i/j swap).
+    /// The tolerance is NOT bit-equality: the SIMD GEMV reduces partial
+    /// sums and fuses products in a different order than this scalar loop,
+    /// so exact bit-equality is not guaranteed by construction.
+    fn scalar_reference(
+        path: &[Vec<f64>],
+        matrices: &[f64],
+        biases: &[f64],
+        state_dim: usize,
+        path_dim: usize,
+    ) -> Vec<f64> {
+        let k = state_dim;
+        let d = path_dim;
+        let mut z = vec![0.0f64; k];
+        for window in path.windows(2) {
+            let delta_x: Vec<f64> = window[1]
+                .iter()
+                .zip(window[0].iter())
+                .map(|(a, b)| a - b)
+                .collect();
+            let mut z_next = z.clone();
+            let mut activated = vec![0.0f64; k];
+            for (i, &dx_i) in delta_x.iter().enumerate().take(d) {
+                if dx_i.abs() < 1e-15 {
+                    continue;
+                }
+                let a_offset = i * k * k;
+                let b_offset = i * k;
+                for row in 0..k {
+                    let mut sum = biases[b_offset + row];
+                    let row_off = a_offset + row * k;
+                    for col in 0..k {
+                        sum += matrices[row_off + col] * z[col];
+                    }
+                    activated[row] = sum.tanh();
+                }
+                for row in 0..k {
+                    z_next[row] += activated[row] * dx_i;
+                }
+            }
+            z = z_next;
+        }
+        z
+    }
+
+    #[test]
+    fn encode_matches_scalar_reference_within_tolerance() {
+        let b = RandomizedSignatureBuilder::new(3, 24, 0x5EED);
+        let path = vec![
+            vec![0.0, 0.0, 0.0],
+            vec![1.0, -0.5, 0.25],
+            vec![0.7, 0.3, -1.1],
+            vec![2.0, 2.0, 2.0],
+        ];
+        let s = b.encode(&path);
+        let expected = scalar_reference(&path, &b.matrices, &b.biases, b.state_dim, b.path_dim);
+        assert_eq!(s.state.len(), expected.len());
+        for (got, want) in s.state.iter().zip(expected.iter()) {
+            let scale = want.abs().max(1.0);
+            assert!(
+                (got - want).abs() / scale < 1e-9,
+                "delegated encode diverged from scalar reference: got {got}, want {want}"
+            );
+        }
+    }
+
+    /// Establishes that `encode` now enforces the ragged-path guard the
+    /// delegated ndarray primitive carries (`assert_eq!`, so it fires in
+    /// release builds too) — a guard this crate's own asserts (which only
+    /// check `path[0]`) did not previously provide. Before the delegation
+    /// this exact input would NOT have panicked; it would have silently
+    /// read out-of-bounds-safe but semantically wrong deltas for the
+    /// shorter/longer point instead. This test discriminates the new
+    /// behavior from the old one, not merely restating the delegation.
+    #[test]
+    #[should_panic(expected = "ragged path")]
+    fn ragged_path_now_panics() {
+        let b = RandomizedSignatureBuilder::new(2, 8, 1);
+        let ragged = vec![vec![0.0, 0.0], vec![1.0, 1.0, 1.0]];
+        let _ = b.encode(&ragged);
+    }
 }