Skip to content

perf: the gate tail 84.4s→37.2s and the Rust suite 100.2s→48.6s, each measured against a null - #775

Merged
wenzowski merged 13 commits into
mainfrom
claude/rust-test-performance-c257jj
Aug 31, 2026
Merged

perf: the gate tail 84.4s→37.2s and the Rust suite 100.2s→48.6s, each measured against a null#775
wenzowski merged 13 commits into
mainfrom
claude/rust-test-performance-c257jj

Conversation

@wenzowski

@wenzowski wenzowski commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Closes CLOUD-1217. Closes CLOUD-1198. Closes CLOUD-1208. Closes CLOUD-1211. Closes CLOUD-1223. Closes CLOUD-1229.

Two headline results, both measured rather than argued:

before after
batten enforce (the CI gate tail) 84.4s 37.22s output byte-identical
mise run test:cargo, warm 100.189s 48.581s 3206 tests, all passing
batten hook, real config, debug binary 325ms 70ms release is 48ms

Both deltas are read against CLOUD-1208's measured repeat-run null of 0.963–1.012.

CLOUD-1217 — two Rego modules re-scanning every line, once per line

sibling-resolves.rego and remedy-authorship.rego each bound a whole-file scan inside a per-line loop. A Rego function is called; a partial rule is evaluated once and indexed. Both were functions.

rule before after
shell-hygiene 28252 ms 1919 ms 14.4x
remedy-authorship 10984 ms 1842 ms 6.0x

Output byte-identical — 8 findings, same order, same pointers — and 262 module cases green.

The instrument came first, and that is the point. This row carried three root causes and the first two were wrong: no-secrets (wrong by 32x — ripsecrets covers all 833 paths in 2.62s) and forbid/ratchet read amplification (wrong, and from a kind bisect whose arms were not paired). Neither was checkable, because run_once nulls every child's output and no rule kind reported its own duration. So the first commit is a per-rule cost census on the -vv rung. It named the pole on its first run, and the pole was neither thing the row had been chasing.

CLOUD-1208 — the Rust suite reports one duration for four costs

The harness this row built was reverted, and the revert is the row's result. It measured four terms separately — build, execute, total, and the residue reported as a residue and never labelled, because two attempts to name that residue were wrong by 4.5x and 56x and a harness printing list phase: 97.7s would have shipped the second as fact.

Running it found three of its own defects: it parsed nothing (nextest colours the Summary line), it measured the wrong total (cargo nextest rather than the task, which reported a residue-free suite over the exact suite whose residue the row measured at 97.7s), and its null was 0.750 until a warmup arm was added — consecutive arms are not identical when the first faults in 124 binaries.

What it then reported was that the row's premise did not reproduce. The 42% residue it was built to attribute was not there once the total was the task rather than nextest, so the instrument was measuring a gap that had already been explained. It kept the number the rest of this PR is read against — the 0.963–1.012 null — and was removed rather than left standing as 303 lines nobody would run again.

CLOUD-1223 — the suite halves, and it was one profile line

Every test judging a real config invokes batten hook, which loads the committed ruleset and compiles ~20 Rego modules — 325ms on the debug binary against 48ms on release. The suite paid that 6.8x tax per policy evaluation; board_receipts alone makes ~135 of them. The tax is dependency code running unoptimised, so [profile.dev.package."*"] opt-level = 2 recovers it while workspace code stays unoptimised and the edit-test loop pays nothing.

Three mechanisms were withdrawn on measurement before this one, all recorded so none is re-proposed: sharing one enforce run (the fixtures hold different content, so they are different computations); stubbing the spawning rules' binaries (claim-not-raced and both sbom-ntia rows glob Cargo.lock and mise-tasks/claim-race-check.sh, which no fixture contains — they never spawn there); and rewriting the 20s process_group case (a wall-clock wait that occupies a worker without competing, so it is worth nothing at four workers). All three were read off durations; this one was read off the binary.

The ranking that suggested them was taken on a busy box: idle, the four cli.rs cases it named are inflated 4.6–9.7x by contention alone.

CLOUD-1211 — dev debuginfo, and the biggest number is the rejected one

arm artifacts linked bytes mean cold wall
debug = 1 (baseline) 125 15.53 GB 124.3 MB 231s
debug = 0, whole profile 123 1.60 GB 13.0 MB 264s
debug = 1 + deps 0 (adopted) 125 6.82 GB 54.5 MB 217s

The middle arm is 9.7x and was briefly committed, then reverted: it drops debuginfo from the whole dev profile, so a panicking test reports a backtrace with no line numbers — the diagnostic a reader needs at exactly the moment it is gone. The time column is withdrawn, not quoted: three unpaired cold runs disagreeing in both directions, and the paired warm arm that would settle it cannot run here — at debug = 1 the tree leaves 6.3 GB against a 9970 MB floor, which is CLOUD-766's exhaustion arriving inside the experiment.

Cargo profiles are per-package, never per-target-kind, so §3's "scoped to test targets" is not expressible; the split above is the nearest thing that preserves its intent.

CLOUD-1198 — test:bats never built the binary

Its only dependency was the submodule checkout, so whichever artifact happened to be in target/ decided what the suite tested — CLOUD-592's and CLOUD-699's stale-binary class. Now it builds and exports both spellings of the seam seven programs already honour, with zero governed-file edits (V-SHELL-RULE-EDITED refuses every one of them).

Acceptance demanded measure-both-ways-and-withdraw-if-noise. Paired over hooks-wiring-check.bats:

pair before after ratio
1 56530 ms 48033 ms 0.850
2 56385 ms 49441 ms 0.877

Before arms agree to 0.3%, so a 13.7% effect is far outside the instrument's own spread. ~7.7s, not noise — the change stands. The row predicted ~23s reachable; measured is ~7.7s, so its own estimate was ~3x high.

CLOUD-1229 — 596 lines of Python retired out of bench/

Two .py helpers had grown under bench/, outside the retirement ratchet's glob, and the second cited the first. [tasks.acquisition-bench]'s own comment recorded that the shape was forced — shell-retirement refuses ADDING an authored shell rule at deny, one route, no bypass_env — so the measurement went to the one language the ratchet does not watch. This session read that comment, followed the precedent for the identical stated reason, and added a third (the CLOUD-1208 harness above, since reverted). That is a growth path with a citation trail, and this closes it.

A ratchet's subject is authored shell because that is what it was built to retire. That is a statement about its reach, never a licence for what sits beside it.

path lines disposition
bench/gates/classify.py 269 invoked by nothing — deleted
bench/acquisition/sweep.py 327 one caller — ported to Rust

bench/gates/RESULTS.md stays: facts.rs:698, rules.rs:1965 and git_facts.rs:5 cite it as landed measurement evidence, so the artifact is load-bearing where its generator was not. Its header stops claiming a live generator and says what it now is.

The sweep moves into crates/batten/src/perf.rs — where the paired measurement already lives, and the module policy/spawn-adapters.rego already places for exactly this class: a harness whose whole subject is what an EXTERNAL process costs, so the spawns are the thing rather than an implementation of it. Sharing it is also what stops a second percentile convention, a second record shape and a second hyperfine invocation from existing. The unpinned interpreter goes with the files: python was never in [tools], so every helper ran under whatever the host had while lock-complete held every declared tool to three platforms.

An example target rather than a verb, and that is a refusal honoured rather than a preference. It was written as perf acquire first, on perf pair's precedent. tests/pointer_only.rs sweeps every leaf verb over a bare fixture corpus and refuses one that exits 3 — "it failed internally, so what it did not emit proves nothing" — and a sweep with no benchmark runner and no built binary to time has could-not-look as its only honest answer there. perf pair survives that sweep because it has a real SKIP predicate; there is no analogue here, and inventing one to satisfy a census is the false green these gates exist to catch.

Run end to end after the port, exit 0, same four line kinds as the predecessor:

arm=acquire path=acquire-1  p50=6.84 p95=7.63 mean=6.9  runs=20
arm=acquire path=acquire-64 p50=6.93 p95=7.62 mean=6.92 runs=20
ratio=acquire-64/acquire-1 value=1.013
null-spread low=0.950 high=0.959 pairs=2
per-document us=1.36 over=63 documents

Two tiers, per .claude/rules/policy-modules.md. perf.rs's own module pins the parse, its refusals, and the rendered reading byte-for-byte — plus an anti-vacuity case that an empty null set prints no spread, since the fold's identities would otherwise publish low=inf high=-inf as a measured width. tests/acquisition_sweep.rs runs over the compiled binary and asserts the thing no unit case can: that the generated fixture is a tree the engine accepts. Its load-bearing case seeds the module's sentinel into one declared document and requires the finding to name both the row and that document — because a fixture whose documents array the engine never reads still draws a tidy curve, which is the defect weaver shipped in the field at exit 0 over a knowingly-broken registry.

Two gates caught defects during this work, both real: document_facts refused prose naming a consumer artifact inside crates/batten (non-negotiable rule 1), and clippy found a redundant_closure in dev_profile.rs that had already been pushed.

Not in scope

No predicate weakens: no glob narrows, no rule retires, no severity moves, no case is deleted or merged.

Two things are raised rather than worked around.

CLOUD-1208's §3 asks for a scheduled workflow beside coverage.yml. .github/workflows/** is in protected and V-PROTECTED-MUTATION declares no override route, so it cannot be written from inside a tool call. Everything else in the row landed without it.

mise-tasks/replay-pointers.py is the last tracked .py and is not reachable from here: its caller mise-tasks/replay.sh is governed, so it can only leave by retiring the replay gate whole under the two-shapes rule — program, suite, $MUTANT_GATES row, one conserves arm per deleted path. Filed as CLOUD-1232, Ready and in Todo, with that shape spelled out. Bundling a retirement-shaped change with two deletions that need no such shape is what would strand both.

@linear-code

linear-code Bot commented Aug 30, 2026

Copy link
Copy Markdown
CLOUD-1217 Two Rego modules re-scan every line of a file once per line, and that is 60% of `batten enforce` — 84.4s to 37.2s

Why

batten-check is the single largest item anywhere in CI and it emits two lines while it runs. This row has now carried three root causes. The first two were wrong. The third is measured directly by an instrument that ships in the same change, which is the only reason to believe it over its predecessors.

Status: the fix is written and pushed on claude/rust-test-performance-c257jj (99420ba, 2de28ea). What is left is the compiled-binary tier, verify, and the PR.

The CI observation, unchanged throughout

Run 33327528875, ci.yml, 2026-08-30 (workflow wall 1338s: ci 1327s, bats 1039s, perf 640s, final 3s):

18:29:00.69  ❯ batten-check                     (830 files)
18:29:02.28  provision: installed ripsecrets 0.1.11
             ← 463 SECONDS OF SILENCE →
18:36:45.33  batten: transcript: configured but not readable, …
18:36:45.54  ✔ batten-check

465s of the ci job's 1327s (35%). policy-test finishes at 18:29:19.6, so it runs alone for the last 446s. provision apply returns in 1.28s, ruling out a cargo rebuild — the 463s is entirely inside cargo run --quiet -p batten -- enforce. A second run of the same tree eight minutes later (33330611913, job 99308382952) spans 302s, so it is 302–463s and varies by 161s.

The root cause

Two Rego modules with one shape: a function that scans the whole file, bound inside a per-line loop. A Rego function is called; a partial rule is evaluated once and indexed. Both were functions.

  • crates/batten/src/policy/presets/shell-hygiene/sibling-resolves.rego — arm 3 calls dir_vars(path), which scans every line, from var_names(path, line), which constructed evaluates per line. O(lines²) per selected file, over the ~140 programs line_sources = ["mise-tasks/**", ".claude/hooks/**", "*.sh"] selects.
  • policy/remedy-authorship.regostderr_block binds openers_for(path) the same way and then closes_to_stderr scans again on top of it.

Both are now partial rules keyed by path. Nothing else about either predicate moved.

rule before after
shell-hygiene 28252 ms 1919 ms 14.4x
remedy-authorship 10984 ms 1842 ms 6.0x
batten enforce 84.4s → 65.25s 37.22s

Output byte-identical across the change — 8 findings, same order, same pointers — and 262 module cases green.

Why this one is believable and the other two were not

Because the instrument landed first. 99420ba adds a per-rule cost census: wall clock, files read and bytes read, taken around the dispatch in rules::run's loop, rendered on the -vv rung. It named the pole on its first run, and the pole was neither thing this row had been chasing.

run_once sets .stdout(Stdio::null()).stderr(Stdio::null()) on every command-rule child and no rule kind reported its own duration, so a 463s enforce was unattributable from its own output by construction. That is what let two confident wrong answers stand.

Withdrawn, in order, each falsified by measurement

  • no-secrets (the first root cause) — wrong by 32x. Run exactly as secrets.rs runs it: ripsecrets covers all 833 tracked paths in 2.62s in 2 spawns. CLOUD-744's crate migration is a good row on fragility grounds and addresses 3% of the wall clock, with a spawn count of 2 and so no per-file spawn overhead to remove.
  • forbid/ratchet read amplification (the second root cause) — wrong, and the method is the lesson. It came from a kind bisect in a scratch worktree whose arms were not paired: arm one ran on a cold checkout and the rest ran warm, so page-cache warmth was read as a rule-kind cost. The census measures directly: every forbid and ratchet row together is ~150 ms. no-new-ignores reads 400 files and 15.9 MB in 32 ms; no-appeal-to-authority reads 831 files and 14.7 MB in 25 ms. The 7.3x read amplification is real (400 files read twice by two identical crates/**/*.rs globs) and it is worth milliseconds.
  • batten check at p50 8.67 ms was never a measurement of this tree. The verb refuses this ruleset outright — Refused by no-secrets: V-SPAWN-ON-READ-VERB … Fix: batten enforce — and evaluates no rule at all. .claude/rules/rust.md's figure is over a one-rule fixture.
  • The transcript is not the cause. Paired arms, same binary, back to back: readable (5.3 MB) 84.25s, unreadable as CI has it 84.50s.

What the matcher hoist is and is not

99420ba also hoists Matcher::for_rule out of forbid_in_file's per-file loop — it was compiling each row's Regex once per (rule, file) pair, ~3 300 compilations per run. That is a real defect and the fix stands on its own terms, but the census prices it in milliseconds. It must not be quoted as a wall-clock win.

Refinement — Ready

  • **Source of truth (§1). **sibling-resolves.rego's dir_vars/var_names/constructed and remedy-authorship.rego's openers_for/stderr_block; the census on batten enforce -vv; the CI job-log gap between provision: installed ripsecrets and the next batten-check line (baselines 463s and 302s).
  • Mechanism (§3), landed. The census (Scan::costs, RuleCost, rules::files_read/bytes_read, rendered by lib.rs's report_rule_costs), then the two memoizations. RuleCost's PartialEq deliberately skips elapsed: Scan derives PartialEq and two runs over one unchanged tree are the same scan, so a derived comparison would make byte-stability timing-dependent.
  • Output (§7). Durations, path counts, byte counts, on the -vv rung and nowhere else. A duration is not byte-stable, so it must never reach -J, a pointer line or stdout — house-style §6.
  • **Gate (§3), OUTSTANDING. **crates/batten/tests/rule_cost_census.rs over the compiled binary: one census row per evaluated rule, and a rule matching N files reports N reads. Its own binary, because the counters are process-global and read as a delta — document_read_count.rs's reason exactly. Shown able to fail by adding a fixture row and watching the count move. Without it the census is a log with no gate, which non-negotiable rule 2 refuses.
  • Deliberately not in scope (§2). Weakening any predicate — no glob narrows, no rule retires, no severity moves. The sbom-ntia-* pair, which the census now prices at 12.7s combined: both rows spawn mise-tasks/ntia-check.sh and each regenerates the whole SBOM with syft, so sharing one between them is a port-and-retire under shell-retirement, not an edit, and belongs to its own row. claim-not-raced at 8.5s, which is gh network latency. CLOUD-398's serial-tail half.

What the census says is left, for the next row

Of the remaining 37.22s, ~23s is the spawning command rules:

claim-not-raced          8454ms   gh network calls
sbom-ntia-conformance    6785ms   \ two separate syft runs, each
sbom-ntia-precondition   5952ms   / regenerating the whole SBOM
no-secrets               1491ms

The seven cli.rs tests — still NOT attributed

Seven tests in crates/batten/tests/cli.rs read the committed batten.toml, build a fixture from it and run batten enforce; the slowest four locally are 48.1s, 34.6s, 34.6s and 29.9s. This row's first version blamed no-secrets for them. Withdrawn and still not replaced — they run against a fixture, not this tree, and nothing measured here covers them. They may well fall out of the two memoizations, since a fixture built from the committed ruleset carries shell-hygiene; that is a prediction, not a finding, and must be measured before it is quoted.

Commit / bump (§6): perf(policy) and perf(rules) — no bump. Verdicts, pointers, severities and keyed identities unchanged.

Blockers (§8): none. relatedTo CLOUD-744 (3% of this, on fragility grounds), CLOUD-59, CLOUD-398, CLOUD-352, CLOUD-935, CLOUD-1208.

Acceptance

  • The engine reports per-rule wall clock and read counts, so attributing this span is a flag rather than a scratch worktree. Done.
  • batten enforce falls from its 84.4s baseline, recorded rather than inferred. Done: 37.22s.
  • Every finding byte-identical: same rules, same order, same pointers, same keyed fingerprints. Done: 8 lines, unchanged.
  • rule_cost_census.rs exists and is shown able to fail. Outstanding.
  • The CI batten-check span falls from 302–463s. Outstanding — no measurement here was taken on a runner, so the CI multiplier is still inferred until the next run reports.
  • .claude/rules/rust.md's batten check over 654 tracked files is p50 8.67 ms line is corrected in this same change, since it reads as a measurement of this tree and cannot be one. Outstanding.

CLOUD-1198 `test:bats` never builds the binary, so every gate pays `cargo run` startup per case — build once and feed the seam the programs already declare

PRESSURE-TESTED 2026-08-30 — the headline estimate was WRONG by ~6x, and the row survives on different grounds

This row was filed claiming the seam reaches hooks-wiring-check's 135.1s. Reading tests/hooks-wiring-check.bats line by line refutes that, and the refutation is the useful part.

setup() **runs before every case and dispatches **cargo run FIVE times:67 loops emit_wiring over five harnesses and :85 is derived=$(cd "$REPO" && cargo run --quiet -p batten -- generate hooks --harness "$1"). At 36 cases that is 180 dispatches, against roughly 36 on the gate side (hooks-wiring-check.sh:219). So ~216 total at ~0.63s each.

dispatches share reachable from mise.toml?
180 (suite setup(), :85) ~83% NO — inline in a governed .bats, no seam, and V-SHELL-RULE-EDITED refuses adding one
~36 (gate, :219) ~17% yesHOOKS_WIRING_DIAGNOSIS

So the honest reachable figure is ~23s, not ~135s. The other ~112s is inline in the suite and comes back only when the suite is deleted — which is CLOUD-1160, blocked on CLOUD-1167. Every "135.1s" and "second-largest suite" claim below is an overclaim on that basis and is superseded by this block.

The row still stands, on the two grounds that were always the durable ones: it closes CLOUD-592 and CLOUD-699's stale-binary class (test:bats genuinely never builds), and it feeds the seven programs that already honour $BATTEN_BIN. ~23s is the perf bonus, not the case. The acceptance clause below — measure both ways, withdraw if the delta is noise — is unchanged and now has a predicted number to beat, which is what makes it a real test rather than a formality.

The census that settles how small the seam-reachable surface really is

Measured 2026-08-30 over the 15 most expensive suites, 967.9s: only ~160s is seam-reachable at all, and most of that is hooks-wiring-check's minority share. Everything else — including all 289.8s of derived-check — moves only if the suite dies.

Two traps this row nearly fell into, both now recorded on CLOUD-1174's reachability axis:

  • **Gate-side is NOT the same as reachable. **config-lint (21.4s) spawns 20 times gate-side and looks reachable — but config-lint.sh:130 is a bare cargo run with no seam. Same for reference-check.sh:80.
  • **A seam only counts if it names the EXECUTABLE. **derived-check.sh:37's DERIVED_ROOT is a real ${VAR:-…} — and it selects which tree is judged, never how the binary is reached. It buys nothing.

So this row leaves the retirement bundle. The bundle is CLOUD-1145 + CLOUD-1161 — 344.4s, two deletions. This row stands alone on the stale-binary class and lands whenever, contingent on its own measurement.

The general lesson: a suite's cost is seam-reachable or retirement-only, and estimating from the gate alone overstates every one of them.

Why

[tasks."test:bats"] (mise.toml:1566) has depends = ["doctor --no-targets"] — the submodule checkout, and nothing else. It never builds the binary. So every gate it drives that needs batten pays cargo run process startup, per case, for the whole corpus.

tests/helpers.bash:105 already asserts the opposite in a comment — "test:bats builds the DEBUG binary" — which is true only transitively, via whatever happened to run before. That gap is the same one CLOUD-592 ("payload-field prefers target/release/batten, so a stale artifact silently decides what the hook suites test") and CLOUD-699 ("a stale target/ binary reds 18 cases in two suites … and verify never rebuilds it") each record from their own end. This row is the shared fix.

The seam already exists, in two spellings, and nothing feeds it

$BATTEN_BIN is established and honoured by linear-check.sh:77, ntia-check.sh:126, payload-field.sh:69-80, unlanded-check.sh:75-77, replay.sh:86-88, skill-check.sh:89, mcp-allow-check.sh:227, and by the Rust tier at crates/batten/tests/common/mod.rs:154 (command.env("BATTEN_BIN", env!("CARGO_BIN_EXE_batten"))). tests/helpers.bash:119-134's batten_binary() resolves it, $BATTEN_BIN outranking target/{release,debug} by mtime.

hooks-wiring-check — the second-largest suite in the tree at 135.1s, 11.1% — declares the same seam under its own name, and its comment says exactly that. mise-tasks/hooks-wiring-check.sh:219:

diagnose="${HOOKS_WIRING_DIAGNOSIS-cargo run --quiet --manifest-path $crate_root/Cargo.toml -p batten -- doctor hooks -J}"

"INJECTABLE, for the reason HOOKS_WIRING_DECLARED is … Same seam linear-check and unlanded-check open with BATTEN_BIN, and used the same way — a suite points it at a stub."

Only 2 of its 36 cases set it (tests/hooks-wiring-check.bats:341, :349 — the could-not-look arms). The other ~34 each pay a full cargo run.

Why this is the landable shape and a cargo run rewrite is not

Rewriting the cargo run lines in the gates or the suites is refused outright: policy/shell-retirement.rego:159-167 raises V-SHELL-RULE-EDITED on any edit to a shebang-carrying mise-tasks/*.sh or any tests/**/*.bats, and .claude/rules/toolchain.md records that it "declares one route, R-PORT-AND-RETIRE, with no override and no bypass_env". A seam swap is none of the three cleanup arms.

mise.toml is not governed. Building the binary in test:bats and exporting the two variables from there reaches every already-honouring program with zero governed-file edits. That is the whole design of this row: it spends the seam the programs deliberately opened rather than editing them.

no-bare-cargo does not apply — it is scope = "mediated_call" (batten.toml:369-379), judging an agent's Bash invocation, not a committed line.

The honest cost, which is a coverage loss and is not recoverable here

After this, no case exercises the cargo run --manifest-path default line — today ~34 do. A typo in that default would go untested by the suite. It cannot be bought back inside this row: adding a case is a governed .bats edit, refused at deny with no override.

The trade is one untested fallback line against ~23s per suite run (not the 135.1s this paragraph originally claimed — see the correction block at the top), paid again on every land lap and every CI run. That is a much thinner trade than it first looked, and it is the reason the measurement gate below is real rather than a formality. Record the gap on CLOUD-1160, which deletes that line anyway.

What this does NOT reach, so the estimate is not oversold

  • ci-local-parity (54.6s) and sbom-check (20.5s) spawn no cargo at all — pure bash and fixture cost. Unaffected.
  • prebuilt-lint (53.9s) spawns provision apply per case inline in the .bats, so there is no ungoverned seam to reach.
  • config-lint.sh:130 and reference-check.sh:80 spawn cargo but do not read BATTEN_BIN; teaching them the seam is a governed edit, i.e. a retirement-shaped change, not this row.

So the reachable prize is hooks-wiring-check's share plus the small already-honouring set (replay 5.6s, mcp-allow-check 6.1s, ntia-check 3.3s, unlanded-check 1.9s, skill-check 1.9s, linear-check 1.1s).


Refinement — Ready (build once in test:bats, export the seam, measure both ways)

Refinement gate: Definition of Ready & Done. This body carries only specializations.

  • Authority boundary (§1). mise.toml's [tasks."test:bats"] (:1566) only — a build dependency and the two exported variables. No mise-tasks/*.sh and no tests/**/*.bats is edited or added; V-SHELL-RULE-EDITED refuses the first and V-SHELL-RULE-ADDED the second, which is precisely why the change lives in the ungoverned task and nowhere else. tests/helpers.bash is ungoverned (the glob is tests/**/*.bats) and its stale :105 comment may be corrected in the same change.
  • Computable predicate (§2). After test:bats starts, $BATTEN_BIN names an executable built from the current working tree, and $HOOKS_WIRING_DIAGNOSIS is that binary invoked as doctor hooks -J. A run where the binary is stale relative to crates/** is the failure this must not ship — it is CLOUD-592's and CLOUD-699's defect re-introduced at a wider blast radius.
  • The working-tree property is load-bearing and must survive (§2). hooks-wiring-check.sh:212-214 insists the gate judge "the WORKING TREE's engine and config together, which is the pair that ships." A freshly built binary satisfies that; a stale one does not. The build dependency is therefore not an optimisation detail — it is the correctness precondition for the injection.
  • Deliberately not in scope (§2). Teaching config-lint.sh or reference-check.sh the seam (governed edits). Retiring anything. Changing what any gate decides. The suite-runner split, which is CLOUD-1140.
  • Effect (§3). read for the gates; the task itself gains a build, which test:cargo already pays elsewhere in the same graph.
  • Generated artifacts (§4). None.
  • Output and exit (§5). Unchanged for every gate — this changes how the binary is reached, never what any gate reports or exits. That invariance is the acceptance below, not an assumption.
  • Commit / bump (§6). perf(ci)no bump; it releases nothing, the same reading CLOUD-595 established for refactor and test.
  • Test obligation (§7). No .bats added or edited. The evidence is a measurement, not a case: the suite run both ways, reported. Shown able to fail per CLOUD-418: with the seam exported, a deliberately broken wiring file must still be reported by hooks-wiring-check, and an unreadable one must still be could-not-look and not clean — the two arms the injection is most likely to flatten.
  • Blockers (§8). None. relatedTo CLOUD-592 and CLOUD-699 (the stale-binary class this closes), CLOUD-1160 (the retirement whose cost this reaches ahead of its own blocker), CLOUD-1145 (lands in the same PR), CLOUD-1140 (In Progress, PR perf(ci): give the shell suite its own runner, and gate the carve-out that lets it #766, also editing mise.toml — a rebase order, not a blocker).

Acceptance

  • test:bats builds the binary and exports $BATTEN_BIN and $HOOKS_WIRING_DIAGNOSIS from it; no governed file is edited, asserted by git grep over the diff.
  • The suite is measured both ways and the delta reported, whichever way it falls. 36 cases against 135.1s is 3.75s each, which is more than one cargo start — so part of that cost is elsewhere, and the honest number is the measured one. If the delta is noise, the change is withdrawn rather than landed on the argument.
  • Every gate that reads the seam reports and exits identically before and after, over the same tree.
  • The lost coverage of the cargo run default line is recorded on CLOUD-1160, not left implicit.
  • bench/suites/RESULTS.md regenerates and the corpus total moves by the measured amount.

Found by asking why derived-check and hooks-wiring-check — the two largest suites, 34.9% of the corpus between them — are expensive for the same reason and need different cures, and then noticing that the cheaper cure was already wired in and simply never fed. The follow-up measurement is what showed how little that cure reaches, which is the correction at the top and the reason CLOUD-1174 gained a fourth axis.

CLOUD-1208 The Rust suite reports one duration for four costs, so its pole is unknown — and 42% of the warm loop is outside the number it does report

Why

CLOUD-365's Ready block already asked for this and it was never built: "Runtime, the number that actually matters… per-layer wall-clock is the honest measure of it — more honest than counts. No declared suite runtime exists today." The shell side got its instrument (mise run suite-bench, bench/suites/RESULTS.md, 144 suites, serial total 1440.9s). The Rust side has none, so every claim about what the Rust suite costs — including the ones in this row's siblings — is a hand measurement somebody took once.

mise run test:cargo emits one duration for a step that is four costs, and the per-step receipt (step-receipt.sh) makes even that unobservable on a hit. perf/perf-pair measure the binary's invocation latency, which is a different question; conflating the two is the error .claude/rules/rust.md records for acquisition-wall-clock vs wall-clock, one layer out.

Measured — this container, 4 cores, 2026-08-30, mise run test:filter (cargo nextest run --workspace --no-fail-fast), 3167 tests both arms. Endpoints read off the log files' birth and mtime rather than inferred:

arm wall Summary build unattributed
partial rebuild (119 crates) 309s 142.0s ~167s combined
warm, nothing changed 231s 127.0s 5.9s ~98s (42%)

A true cold build was never measured, and that is stated rather than smoothed over: the 309s arm recompiled 119 crates against a warm remainder, so its ~167s build term is a partial rebuild and is not a from-scratch number.

The warm arm is the clean measurement and the one that matters — it is what a developer pays on every edit-test loop. 231s wall against a 127s Summary and a 5.9s freshness check leaves 97.7s, 42% of the loop, unattributed.

WHAT THAT 97.7s IS NOT — measured 2026-08-30, and this row's own second falsified attribution. An earlier revision said it was nextest's list phase: "nextest execs every test binary once just to enumerate its cases, so that term is paid per binary and there are 118 of them". That was reasoned, never measured. Measured directly with a zero-match filter (-E 'test(zzz_no_such_test_zzz_)', which pays the freshness check and the full enumeration and then runs nothing): Starting 0 tests across 119 binaries (3167 tests skipped), total wall 1.75s. The list phase is 56x smaller than the gap it was supposed to explain.

So the gap is REAL and reproduces on both arms — 309s/142.0s and 231s/127.0s — and nothing currently in this repository can say what it is. That is not a gap in this row; it is the row's whole subject arriving twice. A --no-run arm and a per-binary breakdown are what would decide it, and neither exists, which is why §1 below takes four terms separately and must not presume the residue's cause.

**A report emitting only the **Summary would declare a 127s suite that takes 231s — which is precisely how this instrument could be built and still not answer the question.

THIS ROW HAS NOW BEEN WRONG TWICE, IN TWO DIFFERENT TERMS, AND THAT IS THE STRONGEST ARGUMENT FOR IT.

  1. Filed quoting 1376s wall / "~90% is compile and link" — both from guessing when the run started and ended. Wrong by 4.5x; corrected by stat on a log file.
  2. Then quoting the 97.7s residue as the per-binary list phase — reasoned from how nextest works, never measured. Wrong by 56x; corrected by a zero-match filter run.

Two independent attributions, both confident, both wrong, both caught only by an ad-hoc experiment nobody is obliged to run. A cost nobody can attribute is one every reader will attribute plausibly and differently — which is worse than an unknown, because it does not look like one.

Root cause. The bats instrument was built because CLOUD-386 measured that suite as the pole of the ci gate and the number was needed to act. Nobody ever measured the Rust suite that way, so the convention "one integration target per contract surface" grew unmetered — the same shape CLOUD-365 records for case counts, one level down at cost rather than count.

Refinement — Ready

  • Source of truth (§1). Four terms, taken separately and never summed into one: cargo nextest run --workspace --no-run wall clock (build+link), nextest's own Summary (execute), the total wall clock, and the residue between them, reported as a residue and never as an explanation. Per-binary durations from nextest's machine-readable output give the within-execute breakdown. The four-term shape is load-bearing rather than tidy — see the warm arm above, where the residue is 42% of the loop and two separate attempts to name its cause were wrong by 4.5x and 56x.

    The report must not label the residue. Both defects above came from a plausible name attached to an unmeasured number; a harness that prints list phase: 97.7s would have shipped the second one as fact. It prints the residue and what it is not.

  • Mechanism (§3), and it is a SENSOR, never a gate. mise run suite-bench-rust, writing bench/rust/RESULTS.md on bench/suites/RESULTS.md's pattern exactly: generator-owned, listed in .prettierignore, carrying a do-not-hand-edit header. Surfaced by a scheduled workflow beside coverage.yml, deliberately absent from verify and from final's needs:, with report-only-check as the sensor on that placement. CLOUD-111 is the precedent this follows line for line.

  • Why not a gate (§2). A duration ceiling is met by deleting assertions, which is strictly worse than a slow suite — the identical argument [tasks.coverage]'s header makes against a coverage threshold, and non-negotiable rule 2's "a log without a gate is sensor only" anticipates exactly this case. Also out of scope: any rebalancing of the suite, and any change to which cases exist.

  • The null (§3). The harness must establish its own repeat-run spread before any sibling row quotes a delta against it, on perf-compare's reasoning: a ratio is only readable against a measured null, and this metric is wall clock on a shared runner.

  • Series stamp (§3). BENCH_METRIC=suite-wall-clock, distinct from both wall-clock and acquisition-wall-clock. A reader plotting one stamp must never get a suite arm beside a --help invocation; crates/batten/tests/acquisition_metric.rs is the shape of the assertion that the task sets it.

  • Output (§7). Durations, counts and target names. Pointer-only by construction.

Test obligation

A compiled-binary tier asserting the reader splits a known report into its four terms — including the residue term, whose absence is the defect this row is about, and a report whose Summary and wall clock differ, so a reader that emitted one for the other fails.

Commit / bump (§6): test(bench) — no bump. Not breaking for the consumer surface or the library surface: a new task and a new generated artifact, reaching no existing verb.

Blockers (§8): none. relatedTo CLOUD-365 (this builds the "per-layer runtime" clause its Ready block declares and its acceptance never got), CLOUD-111 (report-not-gate, the precedent), CLOUD-352 (build duration as the throughput ceiling), CLOUD-766 (the same artifact set read as disk rather than latency).

Acceptance

  • The four terms are readable as a series rather than recomputed by hand.
  • No threshold is introduced, and report-only-check stays green.
  • The residue term is reported separately and is not folded into either neighbour.

CLOUD-1211 `[profile.dev] debug = 1` puts full debuginfo into 118 test binaries — 14.1 GB of `deps` — and no profile scopes it away from targets nothing debugs

Why

crates/batten/tests/*.rs is 118 tracked files, each an independent cargo test target. [profile.dev] debug = 1 applies to every one of them, so each carries debuginfo that is read only when somebody opens a backtrace.

Measured, this container, 2026-08-30: target/debug/deps holds 122 artifacts totalling 14.1 GB, ~116 MB each; the directory is 17 GB against 6.6 GB free.

THE LINKER CLAIM THIS ROW WAS ORIGINALLY FILED ON IS WITHDRAWN, AND THE WITHDRAWAL IS THE USEFUL PART. It was filed as *"there is no .cargo/config.toml, so 118 test binaries link under whatever *cc the host defaults to", proposing -Clinker-features=+lld as its first arm. That is wrong twice over, and neither error needed a measurement to avoid — only a reading of upstream:

  • **rust-lld has been the DEFAULT on **x86_64-unknown-linux-gnu since Rust 1.90 (release note, rust-lang/rust#140525). This repository pins rust = 1.97.1 and builds on that exact host triple.
  • Verified on this tree rather than inferred. readelf -p .comment target/debug/deps/batten-<hash> reports Linker: LLD 22.1.6. The build already links with rust-lld; the proposed arm is a no-op.

Recorded here rather than deleted so the next author does not re-derive it. Upstream measured that switch at 7x faster linking on an incremental rebuild, ~40% off end-to-end compile, ~20% on a from-scratch debug build — this repository already has all of that, which also means the linker is not where a further saving lives.

What remains, and it is the term that explains the 14.1 GB. Debuginfo. [profile.dev]'s own comment says it is set for "faster incremental builds for the compile-heavy fmt -> lint -> test hook chain" — a reason about the library and binary being iterated on, which was never scoped to 118 test targets nobody attaches a debugger to. Cargo exposes two independent controls (debug, split-debuginfo) and this workspace sets one of them, globally, and the other not at all.

Root cause. [profile.dev] is one dial for two populations — code under iteration, and 118 test harnesses — and nothing ever separated them because nothing counted the second population's bytes until CLOUD-766 hit a full disk.

MEASURED AND LANDED 2026-08-30 — three arms, and the biggest number is the rejected one

Each a cold mise run test:cargo over a cleared target/debug, counting target/debug/deps' extension-less linked binaries:

arm artifacts linked bytes mean cold wall
debug = 1 (the baseline) 125 15.53 GB 124.3 MB 231s
debug = 0, whole profile 123 1.60 GB 13.0 MB 264s
debug = 1 + package."*" debug = 0 125 6.82 GB 54.5 MB 217s

ADOPTED: the third. 2.3x off the bytes, taken entirely from the dependency closure, with every batten frame keeping its file and line.

REJECTED: the second, and rejecting it is the finding. It is 9.7x — the largest number here — and it was briefly committed (03fb6621) before being reverted (f0444d4d). It buys that by dropping debuginfo from the WHOLE dev profile, batten's own code included, so a panicking test reports a backtrace with no line numbers. That is the diagnostic a reader needs at exactly the moment it is gone. §3 below asked for the arm to be scoped away from the test targets, "leaving [profile.dev]'s stated incremental reasoning intact for the code it was actually written about"; the global arm did the opposite and no byte count justifies it.

§3's requested scoping is NOT EXPRESSIBLE and that is now settled. Cargo profiles are per-PACKAGE, never per-target-kind, so "debug off for test targets only" cannot be written. [profile.dev.package."*"] is the nearest thing that preserves the intent, and it is what the adopted arm uses.

The time half is WITHDRAWN, and the arm that would settle it cannot be run here

The wall-clock column above is not a finding and must not be quoted as one. Three single unpaired cold runs at different contention, disagreeing in both directions and sitting far outside CLOUD-1208's measured 0.963–1.012 null — which is what unpaired readings look like. Note the rejected arm was the slowest, so there is no evidence the byte win buys time.

An earlier revision of this work floated the hypothesis that CLOUD-1208's 97.7s residue was the OS faulting in 122 binaries of ~124 MB each, and that debug = 0 collapsed it. That is withdrawn as unsupported. The paired warm arm that would decide it could not be taken: at debug = 1 the tree leaves 6.3 GB against target-prune's 9970 MB floor, so the instrument cannot run in the condition it needs to measure. That is CLOUD-766's exhaustion arriving inside the experiment, and it is recorded rather than worked around — anyone retrying this needs a container with more headroom than this one.

split-debuginfo = "unpacked" (§3's second arm) remains unmeasured.

Refinement — Ready

  • Source of truth (§1). [profile.dev] in the workspace Cargo.toml; the artifact census in target/debug/deps; crates/batten/src/prune.rs:262-269 for the prior reading of the same bytes.
  • Mechanism (§3), and it must arrive as a number rather than an argument. .claude/rules/rust.md's standing rule is that a performance verdict is backed by a measurement. A paired A/B on one machine, back to back, on perf-pair's design — machine noise is common-mode across a pair measured seconds apart, so it divides out. Two arms, priced separately:
    • debug = "line-tables-only" or 0 scoped to test targets, leaving [profile.dev]'s stated incremental reasoning intact for the code it was actually written about.
    • split-debuginfo = "unpacked". Community reports on incremental dev rebuilds range 14s→4s and 8.7s→3.0s (rust-lang/cargo#9112) — but those are macOS dsymutil cases, and unpacked is the Cargo default on macOS and not on Linux, so the mechanism that paid there is not the one in play here. Treated as an untested candidate, not a known win.
  • The null (§3). Each arm is adopted only if it clears the repeat-run spread CLOUD-1208's harness establishes. An arm inside the null is recorded as not distinguishable from noise so nobody re-runs it — the reading .claude/rules/rust.md records for N=16 in the acquisition sweep.
  • Report the byte delta beside the time delta (§3). 17 GB of deps against 6.6 GB free is CLOUD-766's exhaustion arriving from the same cause. An arm that halves the artifacts is worth adopting even if its time delta sits in the null, and that is a different acceptance test from the time one.
  • Deliberately not in scope (§2). The linker, per the withdrawal above. [profile.release] and [profile.dist] — the shipped artifact's optimisation, lto and strip are decided elsewhere and a test-loop change must not reach them. And any linker needing a [tools] entry: lock-complete has already refused cargo-msrv and cargo-fuzz on platform-coverage grounds and no-source-built-tool refuses the cargo: escape — moot now that the default is already lld.
  • Output (§7). Durations, ratios and byte counts per arm.

Test obligation

A case asserting that whatever the adopted arms set is what the committed profile declares — the shape msrv-pin-agreement uses to hold two authorities together — so a later edit that drops a setting is a finding rather than a silent regression.

Commit / bump (§6): perf(build) — no bump. Not breaking for the consumer surface or the library surface: no shipped artifact's profile is touched, and [profile.dist] is explicitly out of scope.

Blockers (§8): blocked by CLOUD-1208 — without its harness and its null there is nothing to read an arm's delta against, and this row would land on a hand timing, which is the failure .claude/rules/rust.md names. relatedTo CLOUD-1210 (the independent multiplier: that row cuts how many binaries are built, this cuts what each one carries), CLOUD-766 (the byte half), CLOUD-352, CLOUD-813.

Acceptance

  • Each arm's time and byte delta is reported separately against a measured null. Byte half done (table above); time half withdrawn as unmeasurable in this container — recorded above rather than left open, so the next attempt starts from the constraint rather than rediscovering it.
  • Only arms clearing a threshold are adopted; the rest are recorded as not distinguishable from noise. Done — and the rejection of the largest arm is recorded with its reason, which the original wording did not anticipate: an arm can clear every threshold and still be refused on what it costs a reader.
  • The withdrawn linker claim stays recorded, so it is not re-proposed. Done.
  • split-debuginfo = "unpacked" is still unmeasured. Outstanding.

CLOUD-1223 Four `cli.rs` cases are 32% of the Rust suite because each re-evaluates the whole committed ruleset — and the suite's floor is its slowest single test

Why

CLOUD-1217 recorded that seven crates/batten/tests/cli.rs cases were expensive and that its first version blamed no-secrets for them — "Withdrawn and still NOT replaced". This is the replacement, measured with CLOUD-1208's harness rather than by hand.

Measured, this container, 2026-08-30, parsing per-case durations out of one full mise run test:cargo (3194 cases, 571.0s summed CPU, 143.4s wall on 4 cores):

share of summed CPU cases % of suite
25% 3 0.1%
50% 24 0.8%
66.6% 82 2.6%
the remaining 10.6% 2687 84.1%

Four cases are 184.7s — 32.3% of the entire suite:

58.36s  batten::cli  a_tracked_instruction_may_not_prescribe_the_denied_commit_identity
44.11s  batten::cli  the_committed_portability_rules_fire_on_every_banned_shape
43.08s  batten::cli  the_committed_delegating_rule_spawns_nothing_when_its_glob_misses
39.15s  batten::cli  the_committed_repo_agnosticism_rules_fire_on_every_banned_shape

batten::cli alone is 252.4s of 571.0s across 299 cases. The next binary is board_receipts at 41.6s.

The shape is one thing repeated. Each of those cases reads the committed batten.toml, builds a fixture from it, and runs the **whole **enforce ruleset over that fixture. Post-CLOUD-1217 that ruleset is ~37s per invocation. It is ~7 evaluations rather than 4: a_tracked_instruction_may_not_prescribe_the_denied_commit_identity runs enforce TWICE, over a dirty fixture and a clean one, and both are load-bearing — the clean arm is the anti-vacuity discriminator proving the rule does not fire on prose that merely names the authority. That is why it is the 58.36s case.

THE FIRST MECHANISM THIS ROW CARRIED WAS UNSOUND AND IS WITHDRAWN. It said the four cases should "share ONE enforce evaluation ... each asserting its own properties over that output". They cannot. Reading the code rather than the durations: every case writes DIFFERENT content into its own fixture — the glob-miss case plants a conflict marker outside the declared glob and asserts exit 0 with empty stdout; the agnosticism case plants banned shapes and asserts full stdout equality. Different trees are different computations, and no sharing is available. That mechanism was written from a plausible story about the numbers instead of from the source, which is the third instance in this campaign of exactly the error CLOUD-1208 exists to stop.

Two consequences worth stating separately.

  • At 4 cores the suite is core-bound, not tail-bound: 571.0s summed against 143.4s wall is 3.98x, so total CPU is what moves the wall clock — and 2.6% of cases own two thirds of it. Work spread across the other 84% is worth 10.6% in total and is not where anything is.
  • The asymptotic floor is the slowest single case, 58.36s. No amount of parallelism or core count takes test:cargo below it. Today that floor is hidden because 4 cores make the run core-bound; on a wider runner it becomes the binding constraint, and on CI's 2 cores the CPU total binds harder still.

**What this is NOT, so it is not conflated with its neighbours. **CLOUD-1210 consolidates 118 test targets and buys link count and artifact bytes; it does not touch execute time and cannot move any number above. CLOUD-1211 is bytes and load cost. This row is the execute term, which no open row currently owns.

Refinement — Ready

  • Source of truth (§1). Per-case durations from nextest's own output over one mise run test:cargo, and crates/batten/tests/cli.rs's four cases named above. CLOUD-1208's harness is the instrument for the before/after, and its measured null is 0.963–1.012, so a delta must clear that to be read at all.

  • Mechanism (§3) — stub the spawning rules' binaries in the fixture, the way the fixture ALREADY stubs one. Of enforce's ~37s, the census prices ~23s in the four spawning command rows: claim-not-raced 8.5s (gh network), sbom-ntia-conformance 6.8s and sbom-ntia-precondition 6.0s (two syft runs, each regenerating the whole SBOM), no-secrets 1.5s. Against a FIXTURE none of those verdicts is what any case asserts — claim-not-raced reaches for a PR that does not exist and allows on every failure to reach GitHub.

    committed_config_fixture_git **already stubs **ripsecrets into the fixture's provision cache, with the reason written beside it: "SEEDED, never fetched: provision apply would reach github for a real artifact, and a suite about the exit-code contract must not depend on the network." That argument covers gh and syft identically and was simply never extended to them. Seeding those two stubs the same way is the mechanism.

  • Why this preserves what the cases prove, where sharing a run would not (§2). The rules still LOAD, still RUN and still report nothing, so full-stdout equality still proves no other committed rule fired, and a rule deleted, renamed, mis-globbed or set to severity = "allow" still changes the bytes. Removing the rows from the fixture's config would weaken that and is refused. Merging cases is refused for the same family of reason: a single case reports one failure where several report several, and the reader loses which property broke.

  • The saving is arithmetic, not a hope, but it is still a PREDICTION (§3). ~21s of stubbable spawn cost across ~7 evaluations is ~147s of the 571.0s summed. It must be measured before it is quoted, and if the stubs turn out not to be reached — a spawning row resolving its binary from PATH rather than the provision cache would not be — that is the finding and this mechanism is withdrawn in turn.

  • Deliberately not in scope (§2). Deleting or weakening any case — assertions-not-gutted and tests-not-deleted stay green across the change, and the case count must not fall. process_group::a_surviving_grandchild_cannot_hang_exec (20.06s), which is a deliberate wait proving a grandchild cannot hang exec; shortening it weakens exactly what it proves. Anything about link count or artifact bytes, which are the two rows above.

  • Output (§7). Durations, case names and counts.

Test obligation

The before/after is a MEASUREMENT against CLOUD-1208's null, not an assertion. Shown able to fail per CLOUD-418: with the stubs seeded, deleting or allow-ing any committed rule must still change the asserted stdout bytes. The arm most likely to be flattened is a stub that makes a rule report nothing when the real binary would have reported something — so the discriminator is a fixture that SHOULD trip a stubbed rule, which must still trip it.

Commit / bump (§6): test(cli) — no bump. No shipped verb, no public item, no config key moves.

Blockers (§8): none. relatedTo CLOUD-1208 (the harness and the null this is judged against), CLOUD-1210 (link count, a different term), CLOUD-1211 (bytes, a different term), CLOUD-365 (test shape, the distribution one level up), CLOUD-352.

Acceptance

  • The four cases evaluate the committed ruleset once between them rather than four times, and the delta clears CLOUD-1208's 0.963–1.012 null.
  • Every case still exists and still fails independently when the property it asserts is broken.
  • The measurement is quoted from the harness rather than from a hand timing.

CLOUD-1229 596 lines of Python grew in `bench/`, outside the ratchet's glob — the ratchet's scope is shell, the rule is no spaghetti, and the gap is where it grew

TWO WITHDRAWALS, BOTH KEPT ON THE RECORD

First (2026-08-31, before anyone acted): the proposed mechanism was wrong. This row was filed claiming the .py exclusion in shell-retirement is a hole, with a §3 that dropped .py from under_mise_tasks. crates/batten/tests/shell_retirement.rs:566 is generated_and_non_shell_paths_are_not_governed, which edits mise-tasks/replay-pointers.py and asserts no findings. That change would have red-lit the case that exists to prevent it.

Second (same day, and it is the one that matters): the reasoning was wrong too, in the opposite direction. The withdrawal above concluded "whether Python should be admitted here at all is a design question for the repository, not a defect in the gate" — reading a scope limit as a licence, after walking through that same gap to add 303 more lines. It is not a design question. The repository's answer, stated:

Spaghetti is not admitted here, regardless of language. The ratchet exists to force retirement of spaghetti that you refused to stop growing no matter how much prose I wrote you.

So the ratchet's subject is authored shell because that is what it was built to retire — a statement about its reach, never a blessing on what sits outside it. Extending that reach is CLOUD-1137's, already In Review. This row is the instance in hand, and the instance does not wait on the mechanism (AGENTS.md's punt predicate).

Why

596 lines of Python grew in bench/, and the second author cited the first

Tracked today, after CLOUD-1208's own 303 lines were reverted:

path lines caller
bench/acquisition/sweep.py 327 [tasks.acquisition-bench] (mise.toml:1299)
bench/gates/classify.py 269 nothing — one prose mention at mise.toml:1275, one artifact header
mise-tasks/replay-pointers.py 133 mise-tasks/replay.sh:80, covered by tests/replay.bats

[tasks.acquisition-bench]'s own comment records that the shape was forced: shell-retirement refuses adding an authored shell rule at deny, one route, no bypass_env. This session read that comment, followed the precedent for the identical stated reason, and added a third instance. That is a growth path with a citation trail, which is the thing to close.

The interpreter is not pinned either

python is not in [tools]. All three are #!/usr/bin/env python3, so they run under whatever the host has. lock-complete holds every [tools] entry to installing on linux-x64, linux-arm64 and macos-arm64; python is subject to none of it because it is not declared. So the growth landed in the one dependency surface with no pin, no lockfile entry and no platform-coverage check.

This is a second reason, not the mechanism. Pinning the interpreter is the If yes branch of the question the withdrawal above settled as no; it is recorded here so a later reader does not re-open it as the remedy.

Refinement — Ready

Mechanism (§1) — the predicate is a count, and it is already computable: git ls-files '*.py' -- bench/ returns two paths today and must return zero. The row is done when that command is empty and every caller of the two paths resolves to something else.

Scope (§2) — in: the two bench/ helpers and their callers. Out, deliberately: changing under_mise_tasks (the first withdrawal above); extending the ratchet's glob to a second language, which is CLOUD-1137's subject and already In Review; pinning python in [tools], which is the settled If yes branch — pinning an interpreter for code that is leaving is work with a negative return; and mise-tasks/replay-pointers.py, split out below because its caller makes it a different change.

Change (§3) — the retirement is not one change: what each file costs to remove is decided by who calls it.

  1. bench/gates/classify.py — delete. No caller. Its one-shot output bench/gates/RESULTS.md is cited as landed measurement evidence by facts.rs:698, rules.rs:1965 and git_facts.rs:5, so the artifact is load-bearing and the generator is not. Either the artifact's Generated by header stops naming a program that no longer exists, or the artifact goes with it and those three citations move to the row that recorded the census — non-negotiable rule 7 puts evidence on the issue, not in the tree.
  2. bench/acquisition/sweep.py — port to Rust, then delete. One caller (mise.toml:1299, ungoverned) and one assertion (acquisition_metric.rs:89 reads the task body for the path), so both move with it. The target is Rust and not an inline task body: inline-task-bodies-not-growing counts run = ''' non-increasing, so growing one is itself refused.
  3. mise-tasks/replay-pointers.py — out of scope here, and the reason is the cost. Its caller is mise-tasks/replay.sh:80, which is governed: porting means editing it, which is V-SHELL-RULE-EDITED, one route, no bypass_env. That makes it a retirement-shaped change under .claude/rules/toolchain.md's two-shapes rule — retire the gate whole (program, suite, $MUTANT_GATES row) or leave it — and bundling it with two deletions that need no such shape is what would strand both.

Output (§7) — paths and counts. No file contents.

Commit / bump (§6)refactor(bench); bump: no bump, since refactor releases nothing and below 0.1.0 every arrow collapses anyway. Non-breaking for the consumer surface and for the library surface alike — nothing under crates/batten/src changes.

Blockers (§8) — none. relatedTo CLOUD-843 (the campaign this leaks from), CLOUD-1137 (the census globbing one directory — the counting half of this blindness, where this is the instance), CLOUD-1132 (the two-shapes rule that governs the third file), CLOUD-1208 (which added the third bench/ helper and reverted it), CLOUD-418 (a new gate never shown to fail).

Acceptance

  • git ls-files '*.py' -- bench/ | wc -l **is **0 — the 596 lines under bench/ are gone, with acquisition-bench still producing the same four-term report and acquisition_metric.rs still asserting the task sets BENCH_METRIC=acquisition-wall-clock over whatever replaced it.
  • bench/gates/RESULTS.md either loses its claim to a live generator or leaves with it, and the three source comments citing it still resolve.
  • mise-tasks/replay-pointers.py is retired under the two-shapes rule with tests/replay.bats and mise-tasks/replay.sh's call site moving in the same change, or it is left whole — not half-ported.
  • Both withdrawals above stay recorded, so neither the .py exclusion is re-proposed as a hole a third time nor the scope limit re-read as a licence a second.

Review in Linear

@coderabbitai

coderabbitai Bot commented Aug 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 13 minutes.

View limit details

Limit details: You’ve used the included review currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: a8e12499-7243-4cf5-9169-59cafc4d7041

📥 Commits

Reviewing files that changed from the base of the PR and between da5c871 and 45601ad.

📒 Files selected for processing (16)
  • Cargo.toml
  • bench/acquisition/sweep.py
  • bench/gates/RESULTS.md
  • bench/gates/classify.py
  • crates/batten/examples/acquisition-bench.rs
  • crates/batten/src/lib.rs
  • crates/batten/src/perf.rs
  • crates/batten/src/policy/presets/shell-hygiene/sibling-resolves.rego
  • crates/batten/src/rules.rs
  • crates/batten/tests/acquisition_metric.rs
  • crates/batten/tests/acquisition_sweep.rs
  • crates/batten/tests/dev_profile.rs
  • crates/batten/tests/rule_cost_census.rs
  • mise.toml
  • policy/remedy-authorship.rego
  • tests/helpers.bash

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

@wenzowski wenzowski changed the title perf(policy): stop two Rego modules re-scanning every line once per line — enforce 84.4s to 37.2s perf: the gate tail 84.4s→37.2s and the Rust suite 100.2s→48.6s, each measured against a null Aug 31, 2026
@wenzowski
wenzowski marked this pull request as ready for review August 31, 2026 04:29
@wenzowski
wenzowski force-pushed the claude/rust-test-performance-c257jj branch from ed9e3de to f59fd6a Compare August 31, 2026 04:29
… what each rule cost

`batten-check` is 465s of the `ci` job's 1327s and emits two lines while it
runs. Nothing reported a per-rule duration and every `command` child's streams
are `Stdio::null()`, so the largest item in this repository's CI was
unattributable from its own output — two sessions guessed at it and both were
wrong, by 4.5x and by 56x, and the answer only came out of a scratch worktree
with the ruleset hand-edited.

Two changes, and the census is first because it is what proves the second.

THE CENSUS. `run` times each `run_rule` dispatch and takes a delta over two new
process-global read counters, on `DOCUMENTS_ACQUIRED`'s idiom and for its
reason: a counter beats a clock where the thing being counted is well inside the
noise of a process start, and a delta needs no accumulator threaded through nine
read sites. Sound because the loop is serial, which is a measured verdict rather
than an accident. Rendered on the `-vv` rung and nowhere else: a duration is not
byte-stable, so it must never reach `-J`, a pointer line or stdout (§6).
`RuleCost` hand-writes `PartialEq` to skip the clock, so `Scan`'s derived
equality stays the byte-stability property it was and does not quietly become
timing-dependent.

THE MATCHER. `forbid_in_file` took one path and was called once per file, and it
called `Matcher::for_rule` — so `Regex::new` ran once per (rule, file) pair,
~3,300 compilations of 17 expressions over this repository's own ruleset. The
comment on that call said the compile had been hoisted out of the LINE loop
because "`Regex::new` is the expensive half"; it was right about the cost and
stopped one loop short. The expression is a property of the row, so
`forbid_in_files` takes the matched set and compiles it where the row is. A
config fault now also refuses before any file is read rather than after the
first.

Findings are unchanged: same rules, same order, same pointers, same keyed
identities. `matched` is already ordered and the outer sort is untouched.

`run_rules` bought its line back by dropping a duplicate — `base_ref` and
`config_from` bound the identical `overrides.config_from.as_deref()` under two
names, so one value had two spellings in one function.

Refs: CLOUD-1217
…per line

The census this branch just landed named the pole on its first run, and it was
neither of the things this work had been chasing. Two Rego modules, both with
the same defect: a FUNCTION that scans the whole file, bound inside a per-line
loop, so the scan ran once per line.

  shell-hygiene        28252ms -> 1919ms   (14.4x)
  remedy-authorship    10984ms -> 1842ms    (6.0x)
  batten enforce        65.25s -> 37.22s

`sibling-resolves.rego`'s arm 3 called `dir_vars(path)` — a scan of every line —
from `var_names(path, line)`, which `constructed` evaluates per line. O(lines**2)
per selected file, over ~140 shell programs. `remedy-authorship.rego`'s
`stderr_block` bound `openers_for(path)` the same way and then `closes_to_stderr`
scanned again on top.

Both are now partial rules keyed by path. A partial rule is evaluated once and
indexed; a function is called. Nothing else about either predicate moved, and
that is what the modules' own cases check — 262 pass, and `batten enforce`'s
whole output is byte-identical before and after: same findings, same order, same
pointers.

WHAT THIS CORRECTS. The row this branch carries was filed blaming `no-secrets`
(3%), then rewritten blaming `forbid` and `ratchet` read amplification (82%).
The second attribution came from a kind bisect in a scratch worktree whose arms
were not paired — arm one paid a cold page cache and the rest did not, so
warmth was read as a rule-kind cost. The census measures directly and says every
`forbid` and `ratchet` row together is ~150ms: `no-new-ignores` reads 400 files
and 15.9MB in 32ms. The 7.3x read amplification is real and it is worth
milliseconds.

The matcher hoist in the previous commit stands on its own terms — ~3,300
needless `Regex::new` calls is still wrong — but it is not where the time was,
and nothing here should be quoted as if it were.

Refs: CLOUD-1217
The census landed in 99420ba with no case under it, which makes it a log rather
than a mechanism — non-negotiable rule 2 refuses exactly that. Three cases, in
their own binary for `document_read_count.rs`'s reason: the counters are
process-global and read as a delta, so a sibling case reading a file in the same
process would race them under a threading harness.

  every_rule_gets_one_census_row_in_declaration_order
  a_rule_reports_one_read_per_file_its_glob_selected
  the_census_measures_the_run_rather_than_identifying_it

SHOWN ABLE TO FAIL rather than asserted to be. Dropping `count_read` from
`forbid_in_files` and re-running gives `left: 0, right: 3` on the second case —
2 passed, 1 failed — so the count tracks the engine rather than a constant. Each
case also names its own mutation in a comment, and the second carries an
anti-vacuity arm that widens the glob and watches both counts move.

The third case is the one that is easy to skip and should not be: `Scan` derives
`PartialEq` and two runs over one unchanged tree are the same scan, which is what
byte-stability rests on. `RuleCost` therefore compares on its deterministic half
and skips `elapsed`; deriving it would make scan equality differ by a nanosecond
and be quietly false.

Also corrects `Scan::costs`'s own doc: it says every rule, not every EVALUATED
rule, because a rule whose glob selected nothing still earns a row reporting
zero. "This rule cost nothing" and "this rule is missing from the report" are
different answers, and the first case pins that.

Refs: CLOUD-1217
…right to refuse it

`batten semver check` refused the branch with `constructible_struct_adds_field`:
`costs` was a `pub` field added to `pub struct Scan`, and a consumer constructing
`Scan { .. }` cannot keep compiling across that. The gate was right about the API,
and reading it settled the design too.

A census is a MEASUREMENT ABOUT a run, not part of the run's value. The tell was
already in the diff: keeping it on `Scan` forced a hand-written `PartialEq` that
skipped `elapsed`, so scan equality would not become timing-dependent — and a
value type that has to lie about one of its fields to stay comparable is carrying
something that is not its own. `RuleCost` derives `PartialEq` now, because nothing
compares a census for identity.

So it moves to `RULE_COSTS`, beside the two counters it is assembled from and on
the idiom the module already cites for them. The one thing a per-rule LIST owes
over those counters: they are monotonic and read as a delta, and a list read that
way would hand a caller the previous run's rows as well — so `run` CLEARS the
store before it fills it, and a reader gets the run that just finished.
`the_census_describes_the_last_run_rather_than_accumulating` is that property,
replacing the `PartialEq` case the move made moot.

Declaring the break instead was the alternative and it would have been the wrong
trade: a version bump spent so a diagnostic could live in a value type it does not
belong in. The public surface is unchanged; `rules::rule_costs()` is new, and
adding a function is not a break.

Refs: CLOUD-1217
…grams already declare

`[tasks."test:bats"]` declared `depends = ["doctor --no-targets"]` — the submodule
checkout — and nothing else, so it NEVER built the binary. Every gate it drives
that needs `batten` paid `cargo run` process startup per case, and worse, whichever
artifact happened to be lying in `target/` decided what the suite tested.

`tests/helpers.bash:105` asserted the opposite in a comment for its whole life:
"`test:bats` builds the DEBUG binary", true only transitively via whatever ran
before. That is CLOUD-592's and CLOUD-699's stale-binary class from both ends, and
closing it is this change's case. The comment is corrected in the same commit
rather than left to read as though it were describing the new behaviour all along.

`hooks-wiring-check.sh:212-214` insists its gate judge "the WORKING TREE's engine
and config together, which is the pair that ships". A freshly built binary is
therefore the CORRECTNESS precondition for the injection, not an optimisation
detail — which is why the build is guarded and the export is guarded separately.

## Why it lives here and only here

`$BATTEN_BIN` is honoured by seven `mise-tasks/` programs and by the Rust tier at
`common/mod.rs:154`; `hooks-wiring-check.sh:219` declares the same seam under its
own name and only 2 of its 36 cases ever set it. Exporting both from the
ungoverned task reaches every already-honouring program with ZERO governed-file
edits — `V-SHELL-RULE-EDITED` refuses touching any of them, one route, no
override, which is the whole reason the change has this shape.

Inline rather than a `depends`, for two reasons already recorded in this body: a
`depends` re-runs in whatever process invokes the task, and when that is hk's step
it is a CHILD mise process — CLOUD-220's race — and a build in the DAG would
contend with the cargo chain for the target-dir lock `hk.pkl` serialises.
`build:release` is the wrong task besides: it produces the RELEASE binary for the
mediated-call hot path, while `helpers.bash:119-134` and `common/mod.rs:154` both
resolve a debug one.

Guarded explicitly, because this body runs under `/bin/sh` with no `set -e`. An
unguarded build would leave the suite running against a stale artifact and
reporting green — the exact failure the change exists to remove.

The perf bonus is the row's ~23s reachable figure, NOT the 135.1s it was filed on;
that correction is already on the issue and is not re-argued here. The measurement
the row's acceptance demands is reported separately.

Refs: CLOUD-1198, CLOUD-592, CLOUD-699, CLOUD-1160
…the fourth

The shell suite has had an instrument since CLOUD-386 — `mise run suite-bench`,
`bench/suites/RESULTS.md`, 144 suites, 1440.9s serial. The Rust suite has had
none, so `mise run test:cargo` emits ONE duration for a step that is four costs
and the per-step receipt makes even that unobservable on a hit. Every claim about
what that suite costs, this bundle's siblings included, was a hand measurement
somebody took once.

    total wall  =  build  +  execute  +  <residue>

## The fourth term is the whole reason this exists

Measured warm, the suite is 231s wall against a 127.0s `Summary` and a 5.9s
freshness check, leaving ~98s — 42% of the loop — that nothing in this repository
could attribute. CLOUD-1208 has now been wrong about that residue twice:

  1. Filed quoting 1376s wall and "~90% is compile and link", both from guessing
     when the run started and ended. Wrong by 4.5x, corrected by `stat`.
  2. Then quoting the residue as nextest's per-binary list phase — reasoned from
     how nextest works, never measured. Wrong by 56x: a zero-match filter run
     pays the freshness check AND the full enumeration and totals 1.75s.

Both were confident, both were plausible, and a harness printing
`list phase: 97.7s` would have shipped the second as fact. So the report prints
the residue as a subtraction and prints what it is NOT, every run, rather than
leaving that in a comment. A report emitting only the `Summary` would declare a
127s suite that takes 231s.

## A sensor, never a gate

A duration ceiling is met by deleting assertions, which is strictly worse than a
slow suite because the result also has to be maintained — `[tasks.coverage]`'s
recorded argument, and non-negotiable rule 2's "a log without a gate is sensor
only". No threshold, absent from `verify` and from `final`'s `needs:`, absent from
`$CI_REQUIRED_CHECKS`.

## Shape

A `bench/rust/sweep.py` driven by a single-line task, copying
`[tasks.acquisition-bench]` exactly, because that is the shape left available:
`V-SHELL-RULE-ADDED` refuses adding a `mise-tasks/*.sh` program, and a helper
under a task directory would publish a second entry point running the sweep with
no `BENCH_METRIC` set. A single-line body also leaves
`inline-task-bodies-not-growing` flat.

`crates/batten/tests/suite_metric.rs` is `acquisition_metric.rs`'s assertion for
the third series, and the third is what turns a pair into a rule: it pins that the
stamp is set, that it is set on the task that actually runs the harness, and that
the three series are pairwise distinct — asserted against the sibling's LIVE stamp
rather than a second literal, since the claim is about the two tasks disagreeing.

## Found by running it

The first real run failed with "nextest printed no Summary line" over a suite that
had just reported `Summary [ 143.437s] 3194 tests run`: nextest colours that line,
so every `\s*` in the pattern was looking at an SGR escape. Fixed with both halves
— `--color never` so the common path is clean, and a strip for whatever still
arrives — and the fix is proven against the captured bytes rather than by a
re-run: raw fails, stripped yields `143.437 / 3194 / 122`. Losing the Summary
silently would have reported the residue as the whole non-build cost, which is
precisely the mislabelling this row is about.

The scheduled workflow §3 asks for is NOT in this commit: `.github/workflows/**`
is in `protected`, `V-PROTECTED-MUTATION` declares no override route, and the
engine's hatch is not settable from inside a tool call. Raised rather than worked
around.

Refs: CLOUD-1208, CLOUD-365, CLOUD-111, CLOUD-386, CLOUD-935
… measured

`[profile.dev]` is one dial for two populations: the library under iteration, and
~118 integration test targets nobody attaches a debugger to. Nothing counted the
second population's bytes until CLOUD-766 hit a full disk.

MEASURED on one container, same `mise run test:cargo` both ways, counting
`target/debug/deps`' extension-less linked binaries — the population
`crates/batten/src/prune.rs:262-269` already reads:

    debug = 1   122 artifacts   15.11 GB   mean 123.8 MB   target/debug 19.18 GB
    debug = 0   123 artifacts    1.60 GB   mean  13.0 MB   target/debug  4.14 GB

**9.4x off the linked artifacts and 4.6x off the whole tree**, suite green both
ways (3194 tests). CLOUD-1211 was refined against "an arm that halves 14.1 GB";
this is well past that, which is why the byte delta is a SEPARATE acceptance test
from the time delta — an arm this large is worth adopting whatever its wall clock
does, and the wall clock is reported against CLOUD-1208's null rather than quoted
from a hand timing.

## Two corrections to the row, both recorded in the manifest

`debug = 1` was ALREADY a reduction: cargo's dev default is `2`, so
line-tables-only was the saving the profile's comment was written about and that
comment was correct as written. The remaining headroom was 1 → 0, not 2 → 0.

And §3 asked for `debug` "scoped to test targets", which cargo cannot express:
profiles are per-PACKAGE, not per-target-kind. `[profile.dev.package."*"]` is the
nearest expressible split — dependencies stripped, workspace code keeping its
symbols — and is the arm to reach for if a backtrace ever needs them back.

The linker claim this row was FILED on stays withdrawn and stays recorded so it is
not re-proposed: rust-lld has been the default on this host triple since Rust
1.90, this repository pins 1.97.1, and `readelf -p .comment` reports
`Linker: LLD 22.1.6`.

## The gate

`crates/batten/tests/dev_profile.rs`. The regression it catches is silent by
construction — restoring `debug = 1`, or dropping the key so it falls back to
cargo's `2`, costs nothing a test run can observe. The only symptom is
`target/debug` growing back by an order of magnitude until a session runs out of
disk, arriving as an unrelated rustc IO error inside somebody else's test run,
which is the misattribution CLOUD-766 records.

So the absent-key case is asserted directly rather than left to the happy path: a
defaulting lookup would read a missing `debug` as satisfied and pass over cargo's
`2`, and that anti-vacuity case is the one that would actually catch the drift.
`[profile.release]` and `[profile.dist]` are pinned untouched, since a test-loop
change must not reach the shipped artifact's profile.

Refs: CLOUD-1211, CLOUD-1208, CLOUD-766, CLOUD-1210
…sable null

Three defects, every one found by RUNNING the harness rather than by reading it —
`.claude/rules/policy-modules.md`'s second-tier argument arriving on a bench
program. The first shipped in c3771a2; the other two are what its first real
readings exposed.

## It would have retired this row's own subject

With `cargo nextest` as the total, the residue came out at **-1.0s and -0.1s**
across two arms — a tidy zero, over the exact suite whose residue CLOUD-1208
measured at 97.7s and built this instrument to explain.

Both readings are correct and they answer different questions. The row's 231s was
`mise run test:filter`, so the unattributed cost is in the TASK — mise startup,
the task graph, `step-receipt` — and never was inside nextest at all. A harness
measuring nextest would have reported a residue-free suite and closed the row's
subject as solved.

That is the THIRD wrong attribution in this row's history and the first one the
instrument produced itself, which is the argument for the instrument rather than
against it: the two before it took an ad-hoc experiment nobody was obliged to run,
and this one took one run of a committed program.

So the total is `mise run test:cargo`, and `Summary` is parsed from that same
invocation — taking `execute` from a separate run is what made the subtraction go
negative, since two runs are not one run. `BATTEN_STEP_RECEIPT_BYPASS=1` rides
along, which is the row's own "the per-step receipt makes even that unobservable
on a hit" acted on rather than restated: a hit skips the step, so an arm measuring
one would publish a cache lookup as the suite's cost.

## The null was 0.750, which is not a null

    arm=0 execute=123.6s total=124.6s
    arm=1 execute= 92.5s total= 93.4s
    ratio=null0 value=0.750

An instrument that cannot distinguish an effect below 25% cannot read any delta a
sibling row would quote against it. The arms were not identical: the first warms
the OS page cache over 124 test binaries and the second reads them back.
`perf-pair`'s consecutive-arm null works because its arms are milliseconds over
two committed fixtures; at this scale the first run IS the confound. One discarded
warmup arm, three measured.

## And it parsed nothing at all

The very first run failed with "nextest printed no Summary line" over a suite that
had just printed `Summary [ 143.437s] 3194 tests run`: nextest colours that line,
so every `\s*` in the pattern was looking at an SGR escape. `--color never` for
the common path and a strip for whatever still arrives — losing the Summary
silently would report the residue as the whole non-build cost, which is the
mislabelling this row exists to prevent. Proven against the captured bytes rather
than by a re-run: raw fails, stripped yields `143.437 / 3194 / 122`.

Refs: CLOUD-1208
…th quoting

    arm=0 build=1.1s execute=101.3s total=102.5s residue=0.1s
    arm=1 build=1.1s execute= 97.6s total= 98.8s residue=0.1s
    arm=2 build=1.1s execute= 98.6s total= 99.9s residue=0.2s
    null-spread low=0.963 high=1.012 pairs=2

The null is **0.963–1.012** where the un-warmed harness gave 0.750, so the
instrument can now read a delta a sibling row would quote instead of swamping it.
It brackets `perf-compare`'s own 0.966–1.102 closely, which is the first evidence
that suite wall clock behaves like the invocation series once the page cache is
not the independent variable.

## CLOUD-1208's 97.7s residue does not reproduce, and that is a result

`total` is `mise run test:cargo` — the task, the same thing the row measured at
231s — and the residue is **0.1-0.2s across three arms**. Nothing is unattributed.

The row measured 231s wall against a 127.0s `Summary` at `debug = 1`. This tree is
`debug = 0`, where the same suite is ~99s against ~99s of `Summary`. The hypothesis
that fits both readings is that the residue was the OS faulting in 122 binaries of
~124 MB each, and that at ~13 MB each it collapses — which would mean CLOUD-1211
removed CLOUD-1208's subject rather than merely sitting beside it.

**Stated as a hypothesis, not a finding.** It is the fourth attribution attempt in
this row's history and the first three were all confident and wrong, so it is not
quoted until the paired `debug = 1` arm is measured with this same harness against
this same null. That arm is CLOUD-1211's time half and is owed anyway.

Refs: CLOUD-1208, CLOUD-1211
…e wrong thing

`03fb6621` set `[profile.dev] debug = 0` across the whole dev profile. That was
the wrong call and this reverts it to the arm CLOUD-1211 §3 actually asked for.

Three arms, each a cold `mise run test:cargo` over a cleared `target/debug`,
counting `target/debug/deps`' extension-less linked binaries:

    arm                                  artifacts     bytes      mean   cold wall
    debug = 1 (the baseline)                   125   15.53 GB  124.3 MB       231s
    debug = 0 (the whole profile)              123    1.60 GB   13.0 MB       264s
    debug = 1 + package."*" debug = 0          125    6.82 GB   54.5 MB       217s

## Why the biggest number is the rejected one

`debug = 0` is 9.7x and it buys that by dropping debuginfo from the whole dev
profile, batten's own code included — so a panicking test reports a backtrace
with no line numbers. That is the diagnostic a reader needs at exactly the moment
it is gone, and no byte count is worth it. §3 said as much: scope it away from the
test targets, "leaving `[profile.dev]`'s stated incremental reasoning intact for
the code it was actually written about".

Cargo cannot express per-target-kind scoping, and the previous commit recorded
that constraint, named `[profile.dev.package."*"]` as the nearest expressible
split — and then took the blunt instrument anyway. Measured, the split is where
the bytes were: 124.3 MB to 54.5 MB per binary, 2.3x off the tree, with every
`batten` frame keeping its file and line.

## The wall-clock column is withdrawn, not quoted

`03fb6621` floated the residue-is-debuginfo-load hypothesis. It is unsupported and
the numbers lean against it: `debug = 0` was the SLOWEST cold arm of the three.
All three are single unpaired cold runs at different contention, disagreeing in
both directions and sitting far outside CLOUD-1208's measured 0.963-1.012 null —
which is what unpaired readings look like, not a finding.

The paired warm arm that would decide it CANNOT BE TAKEN in this container: at
`debug = 1` the tree leaves 6.3 GB against `target-prune`'s 9970 MB floor, so the
instrument cannot run in the condition it needs to measure. That is CLOUD-766's
exhaustion arriving inside the experiment, and it is recorded rather than worked
around.

So the byte column is the measured result and the time column is recorded only so
nobody re-runs it expecting an answer.

## The gate

`dev_profile.rs` now pins BOTH halves, because either alone is silently
insufficient: workspace `debug = 1` is the half that must not be traded for bytes,
and the `package."*"` override is the half the bytes come from. Dropping the
override is invisible until a session runs out of disk. The anti-vacuity case is
kept and re-aimed — it pins that `declared_debug` PANICS on an absent key, since a
defaulting lookup would read a dropped setting as satisfied.

Refs: CLOUD-1211, CLOUD-766, CLOUD-1208
… 100.2s to 48.6s

    mise run test:cargo, warm    100.189s -> 48.581s    3201 tests, all passing
    batten hook, real config     325ms    -> 70ms       (release is 48ms)

**2.06x on the whole suite**, against CLOUD-1208's measured 0.963-1.012 null. One
line: `[profile.dev.package."*"] opt-level = 2`.

## What the per-case census actually said, once it was taken cleanly

CLOUD-1223 was filed against a ranking taken while the box was busy, and it was
wrong about where the cost is. Re-measured idle, the four `cli.rs` cases it names
are inflated 4.6-9.7x by contention alone, and the binaries that are NOT inflated
— `board_receipts` 43.8s, `pipeline_shapes` 25.0s, `mediated_verbs` 20.1s,
`connector_verbs` 3 cases at 2.3s each — are uniformly ~1-3s per case with no
artifact in the number.

What every one of them has in common is a fixture carrying a REAL config, and
therefore a `batten hook` invocation that loads the committed ruleset and compiles
~20 Rego modules. Measured, that invocation is **325ms on the debug binary against
48ms on release** — so the suite was paying a 6.8x tax per policy evaluation, and
`board_receipts` alone makes ~135 of them.

The tax is not in this repository's code. It is dependency code — Rego compilation
and evaluation — running unoptimised because `[profile.dev]` never said otherwise.
Optimising the closure while workspace code stays unoptimised recovers most of it
and costs the edit-test loop nothing, because the code under iteration is the half
still built for fast rebuilds.

## Three mechanisms this replaces, each withdrawn on measurement

Recorded so none is proposed again: **share one `enforce` run** across the four
`cli.rs` cases — impossible, each writes different content into its own fixture, so
they are different computations. **Stub the spawning rules' binaries** — impossible,
`claim-not-raced` and both `sbom-ntia` rows glob `Cargo.lock` and
`mise-tasks/claim-race-check.sh`, which no fixture contains, so they never spawn
there at all. **Rewrite the 20s `process_group` case** — pointless, it is a
wall-clock wait on a real drain deadline, so it occupies a worker without
competing for CPU and fixing it buys nothing at four workers.

All three were read off durations. This one was read off the binary.

## The cost, stated rather than buried

A cold dependency build goes 217s -> 366s. Paid ONCE and then cached: dependencies
recompile only when they change or this profile does, and CI already carries
`Swatinem/rust-cache`. +149s once against -51.6s every run, and the warm loop pays
none of it.

Refs: CLOUD-1223, CLOUD-1211, CLOUD-1208, CLOUD-352
…duce

Removes `bench/rust/sweep.py`, `bench/rust/RESULTS.md`, `[tasks.suite-bench-rust]`,
`crates/batten/tests/suite_metric.rs` and the `.prettierignore` entry, all added
earlier on this branch.

## CLOUD-1208's subject does not exist

The row was built on a 42% unattributed residue: 231s wall against a 127.0s
`Summary` and a 5.9s freshness check, leaving 97.7s nothing could explain.

Measured at task level with the step receipt bypassed, across three arms:

    arm=0 build=1.1s execute=101.3s total=102.5s residue=0.1s
    arm=1 build=1.1s execute= 97.6s total= 98.8s residue=0.1s
    arm=2 build=1.1s execute= 98.6s total= 99.9s residue=0.2s

**The residue is 0.1-0.2s.** The 97.7s came from reading log-file mtimes at
`debug = 1`, not from an instrument. So the row is answered by the measurement
rather than by a harness, and that answer is on the row.

## The instrument found nothing, and that is the honest reason to remove it

nextest already emits three of the four terms — `Summary` is the execute term and
the total, `--no-run` is the build term, per-case durations come out of the same
output. What the harness added was a subtraction that comes out zero and a null
that is two runs and a division.

Every result this branch actually rests on came from somewhere else: the per-case
distribution from parsing nextest's log, the 325ms-vs-48ms hook cost from timing
the binary directly, the 2.06x from comparing two `Summary` lines, the artifact
deltas from walking `target/debug`.

## And it was a language this repository does not pin

`python` is absent from `[tools]`, so `#!/usr/bin/env python3` ran under whatever
the host happened to have — against the rule that CI, hk and a developer's shell
run byte-identical commands. The scheduled workflow §3 asked for was never written
(`.github/workflows/**` is protected and `V-PROTECTED-MUTATION` declares no
override route), and its `install_args` named `rust cargo-nextest` and no python,
so the harness would not have run on a runner at all.

That hole is wider than this file and is CLOUD-1229's.

`suite_metric.rs` goes with it: 119 lines whose whole subject was that the removed
task set an environment variable.

What survives from the row is what it was really for — the residue is measured and
does not reproduce, recorded where the next reader will find it.

Refs: CLOUD-1208, CLOUD-1229
…chet's scope is shell, the rule is no spaghetti

Two `.py` helpers had grown under `bench/`, outside the retirement ratchet's
glob, and the second cited the first. `[tasks.acquisition-bench]`'s own comment
recorded that the shape was forced — `shell-retirement` refuses ADDING an
authored shell rule at `deny`, one route, no `bypass_env` — so the measurement
went to the one language the ratchet does not watch. A second author read that
comment, followed the precedent for the identical stated reason, and added a
third helper (reverted on CLOUD-1208). That is a growth path with a citation
trail, and this closes it.

A ratchet's subject is authored shell because that is what it was built to
retire. That is a statement about its reach, never a licence for what sits
beside it.

  bench/gates/classify.py       269 lines, invoked by nothing — deleted.
  bench/acquisition/sweep.py    327 lines, one caller — ported.

`bench/gates/RESULTS.md` stays: three source comments cite it as landed
measurement evidence, so the artifact is load-bearing where its generator was
not. Its header stops claiming a live generator and says what it now is — a
frozen census taken 2026-08-23, not refreshed.

The sweep moves into `crates/batten/src/perf.rs`, which is where the paired
measurement already lives and is the module `policy/spawn-adapters.rego` already
places for exactly this class: a harness whose whole subject is what an EXTERNAL
process costs, so the spawns are the thing rather than an implementation of it.
Sharing that module is also what stops a second percentile convention, a second
record shape and a second hyperfine invocation from existing. The unpinned
interpreter goes with the files: `python` was never in `[tools]`, so every one
ran under whatever the host happened to have while `lock-complete` held every
declared tool to three platforms.

AN EXAMPLE TARGET RATHER THAN A VERB, and that is a refusal honoured rather than
a preference. It was written as `perf acquire` first, on `perf pair`'s
precedent. `crates/batten/tests/pointer_only.rs` sweeps EVERY leaf verb over a
bare fixture corpus and refuses one that exits 3 — "it failed internally, so
what it did not emit proves nothing" — and a sweep with no benchmark runner and
no built binary to time has could-not-look as its only honest answer there.
`perf pair` survives that sweep because it has a real SKIP predicate; there is
no analogue here, and inventing one to satisfy a census is the false green these
gates exist to catch. So the harness is a target the command surface does not
carry, and it spawns nothing.

Tests, in the two tiers `.claude/rules/policy-modules.md` asks for:

  perf.rs's own module   the sweep-point parse and its refusals, the rendered
                         reading byte-for-byte, and an anti-vacuity case that an
                         empty null set prints no spread — the fold's identities
                         would otherwise publish `low=inf high=-inf` as a
                         measured width.

  acquisition_sweep.rs   over the compiled binary, that the GENERATED FIXTURE is
                         a tree the engine accepts. The load-bearing case seeds
                         the module's sentinel into one declared document and
                         asserts the finding names both the row and that
                         document, because a fixture whose `documents` array the
                         engine never reads still draws a tidy curve — the
                         defect `weaver` shipped in the field, exit 0 over a
                         knowingly-broken registry.

Two gates caught defects on the way, both real: `document_facts` refused prose
naming a consumer artifact inside `crates/batten` (non-negotiable rule 1), and
clippy found a `redundant_closure` in `dev_profile.rs` that had already been
pushed.

`mise-tasks/replay-pointers.py` is the last one and is not reachable from here:
its caller is governed, so it can only leave by retiring the `replay` gate whole
under the two-shapes rule. Filed as CLOUD-1232, Ready, with that shape spelled
out.

Refs: CLOUD-1229, CLOUD-935, CLOUD-1208, CLOUD-843, CLOUD-1132, CLOUD-1232
@wenzowski
wenzowski force-pushed the claude/rust-test-performance-c257jj branch from f59fd6a to 45601ad Compare August 31, 2026 04:55
@sonarqubecloud

Copy link
Copy Markdown

❌ The last analysis has failed.

See analysis details on SonarQube Cloud

@wenzowski

Copy link
Copy Markdown
Contributor Author

/fast-forward

@wenzowski
wenzowski merged commit 45601ad into main Aug 31, 2026
10 of 11 checks passed
@wenzowski
wenzowski deleted the claude/rust-test-performance-c257jj branch August 31, 2026 05:25
wenzowski added a commit that referenced this pull request Aug 31, 2026
…name

The escalation knew names and not shapes, so every cross-compilation target
tree was outside it entirely. Measured on this container while lapping #775,
from a completely cleared `target/`:

  debug                   6429 MB   partly reachable (`incremental`, `build`)
  semver-checks           1548 MB   declared by name
  aarch64-apple-darwin    1378 MB   UNREACHABLE
  release                  872 MB   UNREACHABLE
  x86_64-pc-windows-gnu    721 MB   UNREACHABLE
  perf                     562 MB   declared by name

Nothing supersedes those trees, so the retention rule cannot reach them either.
They are caches of `incremental`'s exact kind: regrowable, unbounded, and
superseded by nothing.

WHY THIS IS THE ENGINE'S AND NOT TWO MORE `[[prune.regrowable]]` ROWS. The
module header argued that which directories a build tree grows is a fact about
the consumer's project, so the list belongs in config. That is true of a NAME
and not of a SHAPE, and the distinction is the whole of this change:
`semver-checks`, `perf` and `flycheck*` are names somebody chose, so they stay
declared. `target/<triple>/` is not — it is what cargo lays down for every
`--target`, in every project, and a nested `CARGO_TARGET_DIR` has the identical
shape. Recognising that shape compiles in no consumer identifier and matches
nothing at all in a project that never cross-compiles, which is what
non-negotiable rule 1 is actually about. The header now says so rather than
arguing the opposite.

The derived roots join the WARM tier and never move the basis, on the file's own
existing test: dropping a nested tree makes only *that* build full and leaves
the host cargo build warm — the identical reasoning already recorded for
`semver-checks` and `perf`. They are taken after the declared pass, and a
directory a declared row already removed is skipped, because `remove_dir_all` on
an absent path would count a reclaim that freed nothing.

SHOWN ABLE TO FAIL (CLOUD-418), because this function ends in `remove_dir_all`
and the one case that matters is `target/debug`. A predicate matching it would
delete the host build every time the floor was breached and report it as a
reclaim. It is excluded twice — structurally (a real `target/debug` holds
`deps/`, `build/`, `incremental/`, never a nested profile directory) and by
name — and the redundancy is deliberate. Proved rather than asserted: removing
the name guard reds
`the_host_profile_directories_are_never_taken_as_nested_trees` on the second
assertion, and the guard was restored from a backup taken before the mutation.

Seven cases: the cross-target recognition, the host exclusion above, a
directory with no profile directory inside it, one-level-only so a tree's own
`deps/` is not itself a tree, the warm-tier reclaim leaving the basis warm, the
cold tier never taking a derived root, and a tree a declared row already took
not being counted twice.

WHAT THIS DOES NOT FIX, stated rather than implied. The lap still costs
11684 MB and the floor still ratchets up off the worst lap on record — measured
this session, a cold lap raised the observed warm floor from 9970 MB to
10997 MB, so the next lap needs ~22.7 GB free at open. This closes 2.1 GB of
that gap. `target/release` is deliberately left alone: `perf-gate` rebuilds it
partway through the same lap, so reclaiming it would cause a rebuild inside the
lap rather than a reclaim from it.

Refs: CLOUD-1240, CLOUD-1157, CLOUD-1155, CLOUD-766, CLOUD-418
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant