nexgen: mask-histogram thresholds harvest, E-NXG-1..16, expansion plan v1 - #1176
Conversation
`hdr::words_hamming` and `clam_neighborhood::scent_hamming_distance` each carried their own scalar `count_ones()` fold, bypassing `ndarray_bridge::dispatch_hamming` — the module tree's existing SIMD-dispatched kernel (VPOPCNTDQ -> AVX-512BW -> AVX2 -> scalar, routed to `ndarray::hpc::bitwise` under `ndarray-hpc`). Two implementations of a kernel the crate already owned. `words_hamming` takes `&[u64]`; it now views the words as their byte image before the call. XOR-then-popcount is invariant under byte order, so the same bits are compared regardless of endianness — only traversal order differs. The cast is sound because `u8` has alignment 1 and no invalid bit patterns; a SAFETY comment records this. `words_hamming_sampled` deliberately stays scalar: it compares every `step`-th word, and a strided access pattern cannot route through a contiguous byte kernel. cargo test -p lance-graph --lib graph::blasgraph: 191 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
`typed_graph::masked_traverse` computes the FULL A*A product, then filters every nonzero against a `Vec<bool>` label mask and rebuilds a COO. Whether pushing the mask inside the product could remove real work — rather than only the filter+rebuild overhead — is decided by selectivity, which is measurable before any inward-mask implementation exists. This probe changes no library code: it calls `mxm` directly for the unmasked baseline and `masked_traverse` for the masked result, and reports per-call rows. Two selectivities are kept separate: sigma_result = nnz(masked(A^2)) / nnz(A^2) sigma_columns = |M| / N Their ratio is the concentration factor: >1 means the mask's columns hold more than their share of the product. Measured (n=96, debug): A/dense-random d=0.15 concentration 1.00 / 1.02 / 1.00 B/uniform-sparse d=0.02 concentration 0.99 / 0.55 / 0.89 C/clustered rand-mask concentration 1.00 / 0.99 / 1.03 C/clustered block-mask concentration 0.84 / 0.92 / 0.98 sigma_result tracks sigma_columns in every row: for these distributions the mask removes exactly its share and no more, so mask pushdown would buy the structural mask_density factor and no concentration bonus. Notably a block-ALIGNED mask over a block-clustered graph is inert (0.84-0.98) — the run is paired with a random mask of identical density precisely to isolate that, and community structure alone does not produce concentration. Scope limit, stated because it bounds the conclusion: none of these three generators has degree heterogeneity, so none can produce a concentration effect. A preferential-attachment generator with a hub-correlated mask is the missing arm, and until it is added this probe is silent on that case — it does not falsify concentration, it fails to test for it. `mxm` is O(N^3) over 2 KB BitVec values, so N defaults to 96 to stay runnable in a debug build; SIGMA_PROBE_N overrides it for --release. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
…path-validation-170zcy
…n v1 - .claude/nexgen/harvest/: 11 verbatim read-only Sonnet reports (4 code/doctrine readers; PR sweeps lance-graph #1126-#1175, OGAR #274-#298, ndarray #277-#301) - .claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md: the exposure meter as a nested mask set; T0..T3 entropy shape; 27-row expansion table with a falsifier per row; D-NXG-1..12; probe-first sequencing - EPIPHANIES: E-NXG-1..16 (prepended); AGENT_LOG, INTEGRATION_PLANS, STATUS_BOARD rows; supersession index regenerated last (no diff) No code touched. No public surface removed or demoted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
|
Warning Review limit reachedNext included review available in 58 minutes. View limit detailsLimit details: You’ve used the included review currently available. Your 67 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (21)
Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d876022f-58a9-4b11-9001-92729ecfda9f) |
citation-decay flagged the only new decay in #1176: E-NXG-3 pointed at legacy_outliers.rs:27 with a phrase that sits on the next line. Per the gate's own instruction the fix is a stable anchor, not a corrected number. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
Formatting only, on the inherited sigma-probe example carried by this branch; no semantic change. cargo fmt --all -- --check is clean after this. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf8911026b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let sigma_result = if nnz_full == 0 { | ||
| f64::NAN | ||
| } else { | ||
| nnz_masked as f64 / nnz_full as f64 |
There was a problem hiding this comment.
For sparse graphs with multiple two-hop paths per output—or many probed columns that produce no output—this ratio is not a proxy for work removable by mask pushdown. In the checked GrBMatrix::mxm, every (i, j) is visited and every entry in a.row(i) triggers a lookup, while all successful products for (i, j) collapse into at most one stored nonzero (matrix.rs:206-220). Consequently, workloads with the same sigma_result can have very different masked multiplication costs, making the probe's stop and speedup conclusions unreliable; count column probes/successful semiring operations or time an inward-mask implementation instead.
Useful? React with 👍 / 👎.
| fn mask_blocks(count: usize) -> Vec<usize> { | ||
| (0..count).collect() |
There was a problem hiding this comment.
With the default n=96 and blocks=8, each generated community spans 12 node IDs, but the three counts passed here are 1, 9, and 38. Therefore (0..count) cuts through a community in every row rather than selecting whole communities, so the random/block delta does not isolate block alignment as claimed. Pass the community size into this helper and select complete communities, or use densities whose counts are exact multiples of that size.
Useful? React with 👍 / 👎.
…obe (#1176) Recorded, not fixed: the example is not this session's work and #1176 is merged. Ledger row names both findings (block mask cuts communities; nnz ratio is not multiplication work) and the fix shapes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
The PR went mergeable_state=dirty when #1176 landed a `nexgen-mask-histogram-thresholds-v1` section at the top of STATUS_BOARD.md -- the same insertion point this branch's `bindspace-mailbox-soa-wiring-v1` section uses. Two sessions prepending to one append-only ledger conflict by construction; the resolution is never a choice between them. Both sections survive verbatim, newest write first: D-BSW-0..4 (5 rows) and D-NXG-1..12 (12 rows), each count matched against its own parent. The file grew past BOTH parents (1791 mine / 1786 theirs -> 1811), which is the check that distinguishes a real merge from a side silently dropped. Worth recording because it explains a false signal: GitHub creates no `pull_request` workflow runs while a PR is unmergeable, so the last two pushes produced zero checks and only CodeRabbit's status remained. That looked like a billing cap -- CodeRabbit had reported one minutes earlier -- but a sibling PR got a full run one minute after my push, which falsified it. The conflict was the cause. An absent check is not a passing check, and it is not a broken runner either; read the PR's own mergeable_state before theorising. Index regenerated AFTER the board write, per the #1085 ordering: byte-identical. All four gates re-run locally exactly as the workflows invoke them -- append-only OK (9 checked), no new citation decay since base, dids green on the merge-base-diffed added set, index reproduces byte-identical, zero files under crates/.
2 participants
What this adds (board + docs only, no code in my commit)
.claude/nexgen/harvest/—00-INDEX.md+ 11 verbatim reports from read-only Sonnet agents (4 code/doctrine readers; PR sweeps over lance-graph palette: spend the append margin on the 29 homeless TSV lanes; annotate the 4 overlaps #1126–membrane-tiers: regrade ledger L1 CLOSED, L2 CLOSED-BY-EXISTING-GATE #1175, OGAR fix: F-01 identity-tear race + F-08 bounds check + F-09 poison recovery #274–docs: relive PRs #294/#295/#296 with corrected architecture awareness #298, ndarray plan: unified Foundry roadmap for SMB + MedCare consumers (corrects PR #276 data-model framing) #277–feat(F1): ColumnMaskRewriter with full-tree expression walk + Hash UDF hard-fail #301). Raw evidence, nothing ruled..claude/nexgen/plans/nexgen-mask-histogram-thresholds-v1.md— the Belichtungsmesser reading as a nested mask set (M_1 ⊆ … ⊆ M_16; bucket i =mask_ternlog::<AND_ANDNOT2>; rank = partition point; rollover = popcount test + version-keyed mask split). §2 gives the T0..T3 entropy shape, §3 a 27-row "rooms ahead" table with one falsifier per row, §5 probe-first sequencing. D-NXG-1..12.EPIPHANIES.mdE-NXG-1..16 prepended;AGENT_LOG,INTEGRATION_PLANS,STATUS_BOARDrows; supersession index regenerated last (no diff, see below).What is NOT in this PR
.claude/nexgen/plans/, outsidesupersession_index.py/plan_dids.pyscan paths, so its D-ids are on STATUS_BOARD but invisible to the index coverage column. Stated in the plan header on purpose.Inherited commits on this branch — please read before merging
The designated branch already carried two unmerged commits from another session with no open PR (
ae34721ablasgraph: route two Hamming sites through the shared SIMD dispatch;a11fd64bexamples: sigma-probe for masked_traverse selectivity). Per the branch rules I kept them and mergedmainin rather than discarding or rewriting them. They are not mine and not validated by me (no cargo run in this session). If they should not ship with this, say so and I will move my commit to a fresh branch on your word.Gates run locally
append_only_gate.py origin/main: OK, 9 files checked, four grew (EPIPHANIES 26397→26570, AGENT_LOG 4086→4095, INTEGRATION_PLANS 3199→3214, STATUS_BOARD 1766→1786)citation_decay.py: no input files matchedplan_dids.py: no added plan files under.claude/plans/supersession_index.py: regenerated after the board writes, byte-identical🤖 Generated with Claude Code
https://claude.ai/code/session_016WkNBjHc2e3zuyz9i8qJEv
Generated by Claude Code