C++11 · OpenMPI 4.1.5 · SLURM · Python / matplotlib · 64 ranks, multi-node HPC cluster
A strong-scaling study of a distributed Monte Carlo π estimator across 1–64 MPI ranks on a SLURM cluster, and a point-to-point benchmark measuring a 7.5× latency penalty for messages that cross between compute nodes instead of staying on one.
The scaling study's headline number turned out to be a bug rather than a hardware limit: speedup pinned at exactly 10.0× no matter how many ranks were added, because the work-decomposition loop only had 10 iterations to hand out. Finding that is most of what this project is about.
Runtime vs. rank count for three problem sizes, against ideal T₁/p scaling. The 10⁹-dart curve flattens hard after 16 ranks.
Adding processors to a parallel program does not reliably make it faster, and the reasons it fails to are the interesting part. Three effects dominate at this scale, and each section below isolates one:
- Communication overhead can exceed the work itself. For a small enough problem, coordinating the ranks costs more than the computation being distributed, and adding ranks makes things strictly worse.
- Where a message goes changes what it costs. Two ranks on the same node exchange through shared memory; two on different nodes go over the network. Same API call, very different price.
- The decomposition sets a parallelism ceiling independent of the hardware. If work is divided into N chunks, rank N+1 has nothing to do, and no amount of additional hardware helps.
| # | Component | Focus | Result |
|---|---|---|---|
| 1 | MPI process model | What MPI_Init/MPI_Finalize actually delimit |
Processes exist before MPI_Init; it bounds where MPI calls are legal, not where parallelism starts |
| 2 | Monte Carlo π scaling study | Strong scaling, 1–64 ranks × 10³/10⁶/10⁹ darts | 367 s → 36.7 s; error converges as 1/√N; speedup ceiling traced to a loop bound |
| 3 | Ping-pong latency and bandwidth | Intra- vs inter-node point-to-point cost | 0.63 µs vs 4.71 µs latency — a 7.5× penalty for leaving the node |
| 4 | Subcommunicators and collectives | MPI_Comm_split into a 2D process grid |
Simultaneous row and column reductions, verified on 3×3, 3×4, 4×4 |
| Ranks | Runtime | Speedup | Efficiency |
|---|---|---|---|
| 1 | 367.46 s | 1.00× | 1.00 |
| 2 | 155.83 s | 2.36× | 1.18 |
| 4 | 90.38 s | 4.07× | 1.02 |
| 8 | 73.25 s | 5.02× | 0.63 |
| 16 | 36.76 s | 10.00× | 0.62 |
| 32 | 36.68 s | 10.02× | 0.31 |
| 64 | 36.65 s | 10.03× | 0.16 |
Speedup pinned at exactly 10.0× for 16, 32, and 64 ranks — and that suspiciously round number was the clue. A genuine communication bottleneck produces a curve that bends and then degrades; this one flattened onto a hard ceiling and stayed there, with 32 and 64 ranks matching 16 to within 0.1 s. The cause was ROUNDS, reduced from 100 to 10 to keep runtimes manageable. Work is distributed one round per rank, so with only 10 rounds available, every rank past the tenth received zero work and sat idle for the entire run. The parallelism ceiling was set by a #define in the work decomposition, not by the interconnect and not by Amdahl's law. Reported as saturation, it would have been the wrong conclusion drawn from correct-looking data.
Small problems scale backwards, and the crossover is sharp. At 10³ darts, 64 ranks ran 20× slower than a single rank — efficiency 0.0008. The computation is a few microseconds; the scatter, two reductions, and barrier wrapped around it are not. The same code at 10⁶ darts held 96% efficiency at 4 ranks. Same program, same cluster; only the ratio of work to coordination changed, and it inverted the result completely.
Monte Carlo error depends on total samples and is completely indifferent to how they are distributed. Error tracked 1/√N across every rank count, matching the theoretical convergence rate for independent random sampling, with the 1-rank and 64-rank results at a given dart count landing on the same value. Obvious once stated, but it cleanly separates two axes that share a command line: rank count is a performance question, sample count is an accuracy question, and it is easy to conflate them.
Exchange time vs. message size. The gap between the two curves at small sizes is the cost of leaving the node.
Staying on one node is worth about 7.5× on latency, but the measured inter-node bandwidth came out higher — which is an artifact, not a real effect. Intra-node latency was 0.63 µs against 4.71 µs across nodes, exactly as expected, since shared memory never touches the network. But the linear fit reported 3403 MB/s between nodes versus 2143 MB/s within one, which is backwards. With message sizes capped at 4 KB, both configurations are still latency-dominated, so the slope being fitted is mostly noise. The honest conclusion is that this benchmark measures latency well and does not extend far enough to measure bandwidth at all; the fitted bandwidth figures should not be trusted.
A 2-byte message took 8.5 µs while a 4-byte message took 0.4 µs. The very first measurement in the intra-node run is 20× slower than its neighbours and the effect never reappears at any larger size. It is first-call overhead — MPI's lazy internal setup and cold caches — being charged to the first message instead of to initialization. A warm-up iteration outside the timed region is the fix; without one, the smallest message size silently absorbs the cost of everything MPI does exactly once.
Languages: C++11, C, Python
Parallelism: OpenMPI 4.1.5 — MPI_Send/MPI_Recv, MPI_Sendrecv, MPI_Scatter, MPI_Reduce, MPI_Barrier, MPI_Comm_split, MPI_Wtime
Cluster: SLURM batch scheduling on a multi-node HPC cluster, up to 64 tasks
Analysis: matplotlib, log-log convergence fitting, linear latency/bandwidth regression
Written from scratch: the ping-pong benchmark and its SLURM harness, the MPI parallelization of the π estimator (round-based work distribution via MPI_Scatter, reduction of partial estimates, MPI_MAX timing across ranks), and the row/column subcommunicator decomposition.
01-mpi-process-model/ MPI_Init / MPI_Finalize semantics
02-monte-carlo-pi-scaling-study/ strong scaling, 1-64 ranks
03-pingpong-latency-and-bandwidth/ intra- vs inter-node point-to-point
04-subcommunicators-and-collectives/ MPI_Comm_split, row/column reductions
figures/ plots used above
Each directory has a REPORT.md with the full measurements and analysis, plus the SLURM script and the raw output the numbers came from.
ser_pi_calc.cpp began as a serial dartboard estimator from a public LLNL teaching example — the original attribution is preserved in the file header — and the MPI parallelization is mine. mpi_subcommReduce.cpp started from a commented skeleton with the communicator-splitting logic left to be written.
Two further components were not completed, and no code for them is included: latency hiding with non-blocking and one-sided (RMA) halo exchange, and custom datatypes via MPI_Type_create_struct. RUNNING.md covers what running any of this requires, along with several known issues in the code that I have documented rather than quietly fixed.