perf(mpi): remove the serial O(R·S²) protocol floor from HybridComm

[diagonal-hamiltonian]

What

A partitioned multi-node run was protocol-bound, not network-bound. Every HybridComm
collective rebuilt O(R·S²) integer offset tables, and partition 0 filled all of them
alone while the other S−1 partitions spun at a barrier. Four changes remove that serial
phase; the fifth commit adds the opt-in instrumentation that found it.

Measured with monoprop_COMM_PROFILE=1 (2 nodes, 512-site Hubbard, 4 layers, R=2, S=112),
partition 0's serial fill was 22.33 s of a 26.74 s run while MPI itself was 1.58 s — the
wire was never the problem.

#	change	file
1	Parallel prefix. Block offsets split into a per-(b,t) base needing global knowledge (only O(R·S), stays on partition 0) and a scan over u that partition t owns outright. The count matrix is relaid source-partition-major so each partition writes a contiguous run — dest-major would trade a serial fill for pure false sharing.	HybridComm.h
2	alltoallv_reverse. The answer leg travels the query exchange's legs backwards, so its geometry is the query round's; rebuilding it cost 3·R·S² entries and a barrier for nothing. A ratio, not an identity — the query leg carries Sink::kStride elements per record and the answer leg one, so every reused offset and count is divided by the stride (exact: every forward count is a multiple of it).	HybridComm.h, MPICompat.h, Engine.h
3	Empty-block veto. Each partition publishes a cache-line-padded bitmask of the ranks it sends anything to, and sizing runs source-partition-outer, so a partition that sends nothing costs one load instead of R strided probes. At early layers nearly every block is empty.	HybridComm.h
4	Two-level barrier. PartitionBarrier fans in within an L3 domain then across domains, so the arrival fetch_add and the release store cost O(S/G) coherence transactions inside one L3 slice instead of O(S) across the socket interconnect.	PartitionBarrier.h, CpuTopology.h, PartitionGroup.h

Correctness gate

These changes move no floating-point value — only integer offsets and barrier topology —
so the acceptance criterion is a bit-identical expectation value, not a tolerance. That
held at every step of the ladder and at every rank × partition split.

• 211/211 ctest, plus the comm suite under mpiexec -n 3 and -n 4 (S and R interact
  in the new index arithmetic, so world 2 alone is not enough).
• New test coverage for two gaps. Every pre-existing hybrid_comm case sent the same
  count to every destination per source, which cannot distinguish the
  (rank, dest partition, source partition) index order from its transpose — exactly the
  index change 1 makes. Added two asymmetric-count cases, mutation-verified: an inconsistent
  receiver index makes the old cases pass and both new ones fail. The two-level barrier gets
  6 direct cases (uneven/non-contiguous/singleton domains, poison, reset), because its
  grouped path needs pinning and ≥2 L3 domains to engage through a real ShmComm, which no
  test host is guaranteed to provide.

A/B

Leonardo DCGP, 2 nodes, --exclusive, one allocation for both sides run interleaved (a
per-allocation slowdown would otherwise land on one side and read as an effect). Two venvs built
by the same script from detached worktrees at origin/main and this branch, so benches/ is
identical and only src/ differs. Driver: the in-repo Hubbard model (60 sites / 120 qubits) via
benches/_builders, barriered per layer, expectation value compared as repr — the gate is
bit-identity, not a tolerance.

c1 — 20 layers, atol 1e-4, 1,063,245 terms (small operator: cost set by the per-gate sync count)

layout	main	this PR	speedup	layer 1	⟨O⟩ main vs PR	reps
1 rank × 112 partitions	17.49 s	2.14 s	8.17×	0.911 → 0.100 s	bit-identical	2/2
4 ranks × 28 partitions	3.86 s	2.84 s	1.36×	0.553 → 0.358 s	bit-identical	2/2

c2 — 29 layers, atol 1e-6, 260,928,282 terms (large operator: real work dominates)

layout	main	this PR	speedup	layer 1	⟨O⟩ main vs PR	reps
1 rank × 112 partitions	56.91 s	40.42 s	1.41×	0.916 → 0.090 s	bit-identical	1/1
4 ranks × 28 partitions	31.39 s	31.36 s	1.00×	0.553 → 0.633 s	bit-identical	1/1

Term counts are identical everywhere, and the expectation value is bit-identical between the two
sides at every layout. The two layouts differ from each other in the last bit — the partition
count sets the reduction order — which is pre-existing behaviour these changes do not alter.

What this actually buys, stated plainly:

• The rank/partition layout stops being a trap. On main, putting one partition per physical
  core in a single rank — the natural configuration, and the default — costs 4.5× (c1) or
  1.8× (c2) against splitting the same 112 cores into 4 ranks. With this PR those layouts land
  within 1.36× and 1.00×. That is the O(R·S²) term draining out, and it is the result to read.
• Protocol-bound work gets much faster; compute-bound work at an already-good layout does not.
  Layer 1 at 1×112 is 9–10× cheaper in both configs — it holds a tiny operator, so it measures
  the floor almost directly. Once the operator is 261 M terms and the layout is already 4×28,
  the floor is not what you are paying for and the change is a wash.
• One honest regression: c2 at 4×28 has layer 1 going 0.553 → 0.633 s (+14 %, single
  replicate). Expected in kind — fix 1 adds two barriers per verb, and at S=28 there is little
  serial fill to reclaim. It is why fixes 2–4 are load-bearing rather than optional.
• The effect grows with the world. The removed term is O(R·S²); this A/B is the smallest
  interesting case. On the 8-node / 1024-qubit problem this work was written for, the same four
  changes took the 29-layer run from 541 → 192 s at 1×112 and 163 → 146 s at 4×28, with layer
  1 going 22.7 → 2.4 s. Those numbers come from a separate harness that is not in this PR, so
  treat them as context, not as this diff's verification.

_{A/B binaries built from ca71897; the pushed tip differs from it only by clang-format
whitespace in four files.}

Notes for the reviewer

• mpi is now the largest bucket, ~65k collectives at 25–100 µs, i.e. call latency, not
  bandwidth. Nothing in the table or barrier layer touches it; the only remaining lever is
  fewer collectives per gate (batching commuting gates), which is deliberately out of scope
  here — it touches the evolution scan/fold.

• Change 4 engages only where a rank spans more than one L3 domain, which the profile's
  barrier_groups= field reports rather than leaving to be inferred from timing.

• monoprop_COMM_PROFILE is off by default and allocates nothing then; the hot path pays one
  null check per instrumented region.

• Static analysis. Sonar's reliability gate caught one real defect, now fixed: both new
  transport destructors call CommProfile::dump(), whose std::print can throw, and a throwing
  destructor running during unwinding terminates the process. dump() is now noexcept and
  swallows internally — a diagnostic print must never take the job down. Three families of
  finding are left deliberately unaddressed, and I'd rather say so than quietly churn the diff:
  the memory_order rule wants seq_cst on the barrier's atomics, which would undo fix 4 (and
  fires on the pre-existing flat barrier on main too); the void * payload parameters and the
  [&] capture in alltoallv_reverse mirror the sibling verbs in the same file, where diverging
  would be worse than complying; and the HybridComm field/method counts are pre-existing class
  size that this PR is not the place to refactor.

• One stateful contract is introduced: alltoallv_reverse must directly follow the
  alltoallv_resolve whose layout it reverses. Guarded by a generation counter that throws
  rather than silently reading stale tables, and documented on the verb.

🤖 Generated with Claude Code

[github-actions]

Docs preview: https://pr-166.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (1bd8dc7) to head (fa65466).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #166   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[diagonal-hamiltonian]

🤖 AI text below 🤖

Partition threading: portability fixes, measured

The brief was "simplify the threading, keep performance the same or better, and base it on
profiling". The simplification thesis did not survive measurement; the portability work did, and it
is where the speed-up came from.

What landed

Three independent mechanisms made intra-node threading degrade from system to system:

1. Pinning silently turned itself off under Slurm. enumerate_physical_cores() already filters
   by the rank's affinity mask, then partition_cpusets divided by the node size a second time and
   bailed. Every multi-rank layout on our cluster ran unpinned, which also left the two-level
   barrier inert. The fix measures disjointness between co-located ranks' masks
   (classify_node_mask) rather than mask width — width cannot tell a per-rank slice from a shared
   mask, and collapsing unconditionally would point every co-located rank at the same cores.
2. The barrier spin budget was a fixed iteration count calibrated on Sapphire Rapids, where
   PAUSE is ~140 cycles. On aarch64 cpu_relax() is a ~1-cycle yield, so the same constant gave
   a spin window two orders of magnitude shorter. It is now a time budget (steady_clock,
   injectable for tests, monoprop_SPIN_BUDGET_US).
3. Locality domains keyed on L3 only, so a target without an L3 (A64FX-class ARM) gave every
   participant its own singleton domain — a flat barrier plus S extra cache lines, while
   barrier_groups reported the optimisation as engaged. Domains now come from the deepest shared
   cache, falling back to the NUMA node.

Measured effect of the placement fix alone (monoprop_COMM_PROFILE=1, one node):

layout	barrier_groups before → after	µs/sync before → after
A 1x128 (control, mask = whole node)	32 → 32	24.9 → 21.5
B 8x16 (recommended default)	0 → 4	432–443 → 10.9–19.2
C 2x64	0 → 16	837–844 → 11.5–49.3

End-to-end at ~29M terms on 2 nodes, layout B, interleaved A/B over 4 reps:
energy 2.1× faster, gradient 1.36× faster — non-overlapping distributions, i.e. main's
best run is worse than this branch's worst. inplace is unchanged (0.98×).

Also in: monoprop_BARRIER_GROUPING (so grouped-vs-flat can be measured without unpinning), a
pinned count in COMMPROF (so "believed pinned" and "actually pinned" can differ visibly), an
opt-in monoprop_ENABLE_TSAN build, and MPI-rank coverage for the asymmetric emit gate.

One known regression, not fixed

build_graph at layout A (1×128) is ~1.1× slower, position-matched (main-first 4.76 vs
port-first 5.01; main-second 5.14 vs port-second 6.06). It is one of the four operations that matter,
and the mechanism is unidentified — a spin-budget sweep moves it ≤5%, and layout A is the one layout
where pinning already worked before this branch, so the placement fix cannot be the cause. Layout B,
the recommended default, does not show it. Flagging rather than burying it: I reported this earlier,
withdrew it as non-reproducing, and the withdrawal was wrong — it only became visible after an
A/B-ordering bug was fixed.

What was tried and rejected

• OpenMP is disqualified on correctness, not performance. #pragma omp barrier has no escape
  hatch, so it cannot honour PartitionBarrier::poison() — an engine exception in one partition
  becomes a hang instead of a propagated error. It was also slower where it matters (30 vs 382
  µs/sync at S=32), proc_bind is inert unless set in the environment (and setting OMP_PROC_BIND
  confines the Python thread as a side effect), and with -Wno-unknown-pragmas a missing -fopenmp
  compiles the pragmas away into a silent deadlock.
• TBB: the design is SPMD — the partition index is the communicator rank is the index into
  every shared table — so work-stealing would break the bit-identical-at-fixed-(R,S) contract the
  determinism tests assert.
• The two-level barrier stays. First measurement free of the pinning confound (that is what
  monoprop_BARRIER_GROUPING is for): grouped beats flat at S=128 (build_graph 1.16–1.20×) and
  ties at S=16, with no cell showing flat robustly faster.
• A one-pass leader/follower merge and MPI-4 persistent collectives were both dropped on
  cost grounds, with the reasoning recorded in the branch's results notes.

Why the comm-layer work stopped here

COMMPROF accumulates over the whole process lifetime, so mpi_s has to be compared against the
sum of the operations. Against the sum of the four evaluated operations at ~29M terms on 2 nodes:

layout	evaluated wall	mpi_s	comm share
4×32	13.20 s	1.143	8.7%
8×16	14.02 s	0.419	3.0%
16×8	14.56 s	0.405	2.8%
32×4	17.97 s	0.871	4.8%

At the recommended layout the entire MPI leg is 3% of the work, so perfectly eliminating the
funnel would buy ~3% at this scale. Layout choice is worth more: build_graph is 1.25× faster at
4×32 despite 2.7× worse comm. The robust funnel finding is barrier_peers_s ≈ mpi_s at ratio
1.02–1.07 across every layout — the S−1 non-funnel partitions idle for exactly as long as partition 0
spends inside MPI.

Caveat worth stating plainly: whether the comm share grows with node count is an inference, not
a measurement. A node-count sweep at N = 2, 4, 8, 16 would settle it, and it is the measurement that
justifies or retires the remaining comm programme.

Tests

C++ suite 224/224 green, on x86 and on aarch64 with zero delta. TSan is clean on the three
concurrency suites, and — because hand-rolled atomics give TSan nothing to hook, so "clean" and
"invisible to the tool" look identical — a mutation control demoting each ordering to relaxed one at
a time produces a race report in all five cases. New coverage: partition_cpusets under a
Slurm-style restricted mask, the invariant that co-located ranks stay disjoint, the spin budget
actually reaching the barrier, and the asymmetric emit gate across real ranks (verified at 2 and 4
ranks; a clean skip at 1).

The -L mpi label does not pass, and it does not pass on main either. Running the whole suite
under 2 ranks aborts in shm_comm_oversubscribed_repeated_collectives with terminate called without an active exception — the signature of a joinable std::thread being destroyed, i.e.
thread creation throwing inside the test's own harness loop. Established rather than assumed: it
reproduces 3/3 on the pre-branch commit c1e034c and 3/3 here, it passes at 1 rank in both, and it
is independent of rank binding (--bind-to none included). Everything before it is green — 222 of
224 cases entered on both ranks with zero failures. This is a pre-existing gap in the multi-rank
test path, not a regression from this branch, and it is not fixed here.

Test-suite runtime, and a trap that came with it

CTest runs each Boost case as its own process, so an MPI build pays a full MPI_Init per case, and
MPI_Init initialises every fabric device present even though a single-process test never sends a
message: 8.8 s per process against 0.2 s of user CPU on a login node with 8 HCAs — 34 minutes for
224 cases. monoprop_TEST_EXCLUDE_MPI_FABRIC (default on) skips fabric init and takes the suite to
5.4 minutes.

It is scoped to the single-process serial variants and deliberately kept off the multi-rank ones,
because OMPI_MCA_pml=^ucx makes a 2-rank run of zero_cutoff_upper_atol_zero_is_exact_World hang
indefinitely where it otherwise passes in 29 ms — again on main as well, so it is component
selection rather than engine code. My first version of this applied the setting to every variant and
would have shipped that hang; the scoping seam is a new SERIAL_ENVIRONMENT argument to
discover_tests.

Three developer traps are now written down in AGENTS.md and the docs, because each fails by
producing a plausible result rather than an error: pytest's fd-level capture discards the engine's
fd-2 diagnostics on a passing test (so monoprop_COMM_PROFILE reported nothing and it read as "no
difference between the arms" — just bench now passes -s; measured 0 COMMPROF lines without it
and 6 with), uv sync does not relink the C++ test binary, and a slow CTest run on an MPI build is
MPI_Init rather than slow tests.

[diagonal-hamiltonian]

🤖 AI text below 🤖

Merged main in — and it needed more than conflict resolution

This branch was CONFLICTING with main, which is why it had been getting no CI at all: the
pull_request workflows check out refs/pull/166/merge, GitHub cannot recompute that ref while the
merge conflicts, so no runs were created. The single green check was Label PRs, the one workflow
using pull_request_target (which reads the base ref instead). The merge ref was stale at
7c7994e, parents 5836a06 + c1e034c, dated to the second of the last real CI runs.

Merged as a merge commit rather than a rebase, so the 11 commits already on this PR keep their
hashes and nothing is force-pushed.

Four files conflicted; the rest auto-merged. Two were plain unions of independent additions
(AGENTS.md, parallelism.mdx).

The one that mattered: #208 reinstated the bug this branch fixes

#208 replaced the custom /sys topology discovery with hwloc. That is the better mechanism and it
is kept wholesale — but it also brought back the placement bug this branch exists to fix, because
the guard lives in the placement policy while the affinity filtering lives in discovery, and
rewriting one does not see the other:

// topo_detail::placement_order
if (cores.empty() || group_count * n > cores.size()) { return {}; }

enumerate_physical_cores() has already filtered by the calling thread's affinity, so under
srun --cpu-bind=cores a rank's cores list is its slice — and dividing by the node-wide
group_count a second time asks for more than the slice ever held. The guard refuses, the rank runs
unpinned, and the two-level barrier loses its domains with it (cpuset_domains derives them from
the placement). That is the mechanism measured at 437 µs/sync against 15.5 µs/sync placed.

So rather than keeping either side's file, this keeps main's hwloc discovery and re-applies the fix
on top of it:

• NodeMask, classify_node_mask() and this_thread_cpumask() return, with the PerRank collapse
  in partition_cpusets() so placement_order() stays pure and hardware-free the way refactor: 🧹 use hwloc instead of custom topology discovery #208
  factored it.
• CpuMask replaces cpu_set_t as the exchanged type, since main's CpuSet is now a single PU
  rather than a mask. Fixed-size POD because PartitionGroup ships it through MPI_Allgather as
  MPI_BYTE; hwloc-free so classify_node_mask() is unit-testable without live hardware. 4096 bits
  rather than glibc's CPU_SETSIZE of 1024 — a PU index past the mask is invisible to the
  disjointness test, which would misread a per-rank split as shared.
• pin_this_thread() reports whether the affinity took, which CommProfile's pinned count needs
  to tell "nothing was pinned" apart from "one domain per rank" when barrier_groups is 0.
• cpuset_domains() returns, so the barrier keeps deriving domains from the placement.

Verified on live hardware, not argued from the code

Same live mask, same call, only the classification differs. "as Shared" is exactly main's
behaviour, since main has no NodeMask and always divides:

binding	classified	as Shared (== main)	as PerRank (merged)
--map-by slot:PE=4 --bind-to core, n=4	PerRank	0 placed	4 placed
PE=8, n=8	PerRank	0 placed	8 placed
--bind-to none, n=4	Shared	4 placed	4 placed

The last row is the regression guard: the collapse must not change the shared case, or every
co-located rank would pin to the same cores — worse than not pinning. Reproducer is
maskprobe.cpp (~50 lines); it compiles CpuTopology.cpp in directly, because libmonoprop.so
builds with hidden visibility and does not export the detail::partition symbols.

AGENTS.md now records the trap under Architecture Overview, including that it has regressed
once, so a future rework of that layer has to re-check it.
cpu_topology_policy_per_rank_slice_starves_without_collapse pins the mechanism deterministically,
with no live hardware needed.

Gates on the merged tree

hwloc 2.12.1 is already provided transitively by foss/2025b, so hpc/deucalion/env.sh needs no
change.

• ctest -L serial: 224/224 passed, 0 failed, 412 s
• Fabric-exclusion scoping still correct after main touched cpp/tests/CMakeLists.txt:
  224/224 serial carry it, 0/1 mpi do (the mpi variant must not — OMPI_MCA_pml=^ucx hangs
  2-rank collectives)
• cpu_topology: 14/14 cases, 185 assertions. Per-case assertion counts confirm the two live
  restricted-mask cases asserted rather than early-returning (5 and 6), and that hwloc
  enumerated all 128 cores (129 assertions in the mask-coverage case)
• rank_count_matches_under_an_asymmetric_emit_gate and
  zero_cutoff_upper_atol_zero_is_exact_World: pass at 1, 2 and 4 ranks, 24 assertions each
• Python suite: 609 passed, 0 failed (-m "not mpi"), against a freshly rebuilt editable
  extension
• partition_* + shm_comm_* + hybrid_comm_* together: 36 cases, 5779 assertions, all passing
  — this is the set carrying the bit-identical-at-fixed-(R,S) determinism assertions and the
  barrier poison/reset cases, which are what a placement change most endangers

One note on that last line, because the first attempt looked alarming and the cause is worth knowing
for anyone running the suite on a login node. It reported 291 failed with
RuntimeError: Resource temporarily unavailable, which is EAGAIN from thread creation, presented
via a _BoundSimulatorAdapter repr that itself raises RecursionError — nothing in it points at
threads. ulimit -u is 2,061,974 and looks innocent; the actual ceiling is the cgroup pids
controller, /sys/fs/cgroup/pids/user.slice/user-<uid>.slice/pids.max = 150, against
pids.current ≈ 74 for a shell plus VS Code server. The default heuristic asks for one partition per
physical core = 128 threads, so it cannot start. monoprop_PARTITIONS=4 gives 609/609 and runs 3.5×
faster (52 s vs 179 s). ctest -L serial is unaffected because those cases pass explicit small
partition counts rather than using the heuristic.

I checked whether the same limit explains the shm_comm_oversubscribed abort below. It does not:
that case aborts at 2 ranks under both --bind-to core (S=8, ~16 threads) and --bind-to none
(S=64, ~128 threads) — an 8× swing in demand with no change in outcome — and in isolation with only
76/150 pids in use. So it stays unexplained rather than being quietly attributed to this.

The first CI run found one real thing

Lint failed on cpp/tests/mpi_distributed_layer_equivalence.cpp: prek pins
mirrors-clang-format at v21.1.0, which makes different line-break choices than v20 in two
BOOST_TEST_MESSAGE / BOOST_CHECK_MESSAGE chains. Whitespace only, fixed in fa65466, and the
whole repo now passes under the pinned version. Worth noting that this defect predates the merge
— it sat in the 11 commits undetected precisely because the conflict was suppressing every
pull_request workflow.

Unchanged by this merge, still open

• shm_comm_oversubscribed_repeated_collectives aborts under 2 ranks with terminate called without an active exception — on main as well as here (3/3 both, passes at 1 rank, independent of
  binding). So ctest -L mpi remains unusable as a gate; gate on -L serial plus targeted
  mpirun runs.
• Layout A (1×128) build_graph ~1.1× slower, position-matched, mechanism unidentified. One of the
  four priority operations.
• The N = 2/4/8/16 node-count sweep that would justify or retire the remaining comm programme.

[sonarqubecloud]

Quality Gate passed

Issues
33 New issues
0 Accepted issues

Measures
0 Security Hotspots
48.9% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud