RQ-61-CICAP (#1062) increment 1: retarget ONE non-required job to the idle rust-cpu pool

[avrabe]

Refs #1062, #242. One runs-on line.

Measured, not argued

Four PRs opened within ~30 minutes each reached exactly 3/9 required contexts within minutes and then stalled for hours. The three that completed were the same three every time — Format, Version Pin Sweep, Rivet Validation — and those are precisely the three already on self-hosted. The six that stalled are the six on ubuntu-latest.

Same commits, same moment, same queue. Not job cost (Format is not cheaper than Clippy) — entirely which pool the job asks for.

runs-on census:  56 ubuntu-latest · 4 rust-cpu · 2 light · 1 macos
org runners:     12 online, 4 busy (OTHER repos), 8 IDLE (7 rust-cpu, 1 light)

Why this job

synth-provenance-v1 reconciliation gate needs only dtolnay/rust-toolchain — no apt, no Nix, no cargo install, no network package fetch. That is the same shape as Rivet Validation, which already runs on rust-cpu successfully. This move tests scheduling, not toolchain.

Why not a required job first, even though the required six are what actually blocks

A required context naming a label no online runner satisfies does not fail — it never runs, and a non-running required check deadlocks every merge in the repo. A non-required job that fails is a red someone can read and revert; a required job that never starts is a repo that cannot merge its own revert.

One job per PR, each with its own green cycle, in the risk order posted to #1062:

next	needs	risk
Claim Check	checkout + python only	none — no toolchain at all
Clippy	rust-toolchain + cache	low
Kani	cargo-install of the verifier	medium — network + disk
Test / Z3	sudo apt-get	only after apt is confirmed on the image
Bazel	nix + setup-bazel	last or never — an environmental failure there looks exactly like a real proof failure

The cost being paid, measured this wave

Queue latency converts directly into wasted CI cycles. Two PRs accumulated green against a base that moved underneath them (#1096 at 8/9, #1099 at 9/9) — GitHub re-runs checks when the branch moves, never when the base does. Both had to be rebased and re-run from scratch; #1096 twice.

ci.yml ubuntu-latest 50 → 49; all workflows 56 → 55. Nothing else changed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[avrabe]

The moved job ran on the intended pool and passed — and the timing complicates the thesis, so here it is in full.

job: synth-provenance-v1 reconciliation gate (#396)
runner_name:  pulseengine-ci-01-10          <- one of the idle rust-cpu boxes from the census
labels:       [self-hosted, linux, x64, rust-cpu]
runner_group: hetzner-private
conclusion:   success

Measured against main's ubuntu-latest run of the same job

where                    queue wait     exec     total
branch rust-cpu                 3s      67s       70s
main   ubuntu-latest            4s      37s       41s

The self-hosted runner is nearly 2× slower to execute, and the queue wait was effectively equal. On this single measurement the move made this job worse.

That is not an argument against the change, but it does correct what the change is for:

• The queue wait was 4s on main because the queue happened to be empty at that moment. The stall I measured earlier — four PRs each frozen at exactly 3/9 for hours — happened under contention, when a wave of PRs put ~50 jobs against ubuntu-latest at once. This run does not test that scenario and should not be read as if it did.
• The benefit is contention relief on the required path, not per-job speed. Moving a non-required job off ubuntu-latest costs that job ~30s and frees an ubuntu-latest slot for the six required jobs that actually block merges. That is the trade being made deliberately.

What this changes about the plan

It argues for moving less, not more. Since each move costs execution time, the right target is only what relieves contention on the required path — not a blanket retarget of all 44 non-required ubuntu-latest jobs. The next steps in #1062's ranked order stand, but each should be justified by contention relief and re-measured, and a move that neither relieves contention nor runs faster should be reverted rather than kept for tidiness.

I would rather record this now than let "increment 1 was green" become evidence for a conclusion the numbers do not support.