Give the board one denominator, and name the layer nobody has measured - #192
Open
defenwycke wants to merge 11 commits into
Open
Give the board one denominator, and name the layer nobody has measured#192defenwycke wants to merge 11 commits into
defenwycke wants to merge 11 commits into
Conversation
The levers on this board are quoted in units that do not compose -- execute cycles, one-card wall, per-chunk factors, block-level bounds -- so no one can say whether any of them reaches a stated target. This adds the target (16 L40S), the denominator every lever must be priced in, and the arithmetic that follows. The arithmetic is uncomfortable and it is the point: all known zero-fidelity levers, multiplied, take a block from 18.2 min to 16.6 min against a 10.0 min target. Two corrections come out of E7's evidence rather than from argument: - The GPU is NOT 65% idle. Two full po2-22 proves measure 91.5% and 91.7% mean utilisation, with the only zeros in the first 8 s (execute). E6's ceiling drops to ~1.09x and E9 -- a fork of a dormant upstream, carrying a CUDA deadlock we have already hit -- should close by measurement. - `nvidia-smi` reports kernel residency, not efficiency, so that says nothing about what the card is doing while it is busy. Nobody has ever asked: `nsight`, `ncu`, `nsys` and `occupancy` return zero hits across the whole repo. Which reframes the loudest signal on the board. Three architectures land within 9% of each other across a 4x difference in memory bandwidth. That is not a fact about cards, it is a fact about our kernels -- and kernel work moves neither `METHOD_ID` nor fidelity, in a `risc0-sys` we already vendor against an upstream that is dormant. So the method is a four-layer profile that must reconcile to the same 399 seconds, and a rule that ranks levers against the profile that will exist after the others land -- the miss that made the packer refit a late discovery and leaves the aggregate, 43% of post-#139 cost, still unprofiled.
`agg_chunks()` verifies every chunk receipt against `METHOD_ID` before it reaches the execute-mode branch, so execute mode still needs sixteen receipts from the current guest -- which needs a GPU. Checked rather than assumed: the only sixteen-receipt set we hold is `b62d2a60`, superseded on 2026-08-24; the stale dist binary is `4722cec8`; main and v0.19.0 are `1d6c3792` and no receipts exist for it. `HAZYNC_AGG_EXECUTE` has never been run by anyone, and it moves to the card day, where the session's own chunk proves supply exactly the receipts it has been blocked on. Arithmetic cannot stand in for it either. The recorded `~1,137 M` aggregate cycles come to 1,162 s of the 1,566 s total at the measured rate, but the same composition on the pre-re-baseline `3,636.4 M` predicts ~3,500 s against a measured total above 3,300 s -- it overshoots the whole run. Aggregate segments do not prove at the chunk rate, so the validation/resolution split stays UNKNOWN until the run happens. What replaces it tonight is better anyway: the block's real cycle count. `14.34 G` is MODELLED, `FLEET_SIZING.md` §4 lists that as reason #3 not to trust the card count, and every figure in §1 and §2 divides by it. And one risk to this document was checked and holds: the 1,565.9 s aggregate is current, not stale. The re-baseline that made the aggregate 3.20x cheaper landed in `2d9a636`, and `git merge-base` confirms the v0.19.0 tag contains it.
…nnot fail `FLEET_SIZING.md` §4 lists "14.34 G is modelled" as reason #3 not to trust the card count, and every figure in the sixteen-card plan divides by it. Measured now, on a laptop with no GPU and no chunk receipts: **14.057 G**, so the model was +2.0% high. Chunk work on one L40S is 14,367 s rather than 14,656, the one-card total is 15,933 s, and the bar for sixteen cards moves 1.95x to 1.92x. The conclusion is unchanged, which is the useful part -- the denominator now rests on a measurement. Two things fell out of the same run. Cost-packing is worth **1.18x on the slowest chunk** -- which is what a block's wall-clock actually is -- for a total-cycles cost of +0.06%. That trade was argued from a model; it is measured now, and `reproduce/METHOD_ID`'s "slightly MORE" is quantified. And the packer's own balance metric is computed from PREDICTIONS. It prints 1.00x for a partition that measures 1.059x, so it reports perfect balance whatever the guest does. The per-chunk error is signed rather than noisy and it tracks payload size: the three highest-byte chunks measure -12.3%, -12.7% and -6.2% against prediction, so the byte term is still over-charging after #136 and #137 -- the fourth time a stale packer coefficient has done measurable harm. Neither is a lever on the sixteen-card scale, but #190 is adding a curve dimension to this packer and the refit is a #139 prerequisite worth 2.36x, so these 32 measured points are what it should be fitted against.
…is per-input Workstream B, following the observation that this project's three largest wins were all in the plumbing and none in consensus code. `write_chunk_inputs` coalesces consecutive inputs sharing a `tx_idx` and writes each transaction and prevout blob ONCE per group. `input_costs` prices the same work for EVERY input at full transaction size. Summed over block 962,000 the per-input charge counts 53.40 MB where each transaction counted once is 1.53 MB -- 34.9x apart -- and the measured consequence is already on record: the three highest-byte chunks come in 12.3%, 12.7% and 6.2% under prediction, and the predicted straggler reads 1.001x against a measured 1.059x. It would be wrong to call this a stale constant, and I nearly did. `git log -S` puts the last change to `COST_PER_INPUT_BYTE = 6` in the same commit that introduced grouping. The value was chosen with grouping in view. The defect is a missing dimension. The byte term carries marshalling, which grouping made per-GROUP, and `input_bind`'s re-hash, which is still per-INPUT, and one constant cannot express both scalings -- the comment above it already says ~2.5 of the 6 is the re-hash. So the fix is a second term rather than a refit, and it is the same shape of defect as the missing type dimension #190 is adding a curve dimension for. Teach the packer both at once. The re-hash itself prices at ~133 M cycles, 0.95% of the block -- small, though 87% of those bytes sit in 10% of the inputs, so it lands on the straggler rather than spreading. One thing to stop quoting until it is re-derived: `pack_after_139.py` amortises the byte term across a transaction's inputs while the real packer charges it in full per input. That script is where the 2.36x packer-refit figure comes from.
…alisation `host vb-stages` on the current guest, block 962,000, execute mode, no GPU. Of a 3,223 M-cycle run, reading the witness is 2,519 M -- 78.2%. Everything else together is under 22%: output leaves, the in-block coin map, the entire input loop, utreexo deletes and adds, the merkle root, wtxids. The witness is 7,256,592 bytes, so the read costs 347 cycles per byte. That is #136's finding sitting unfixed in the path #136 did not touch. #136 measured `env::read` of the CHUNK payload at ~147 cycles/byte -- "50.9% of a chunk's entire cycle count before any Bitcoin logic ran" -- and replaced it with `write_slice`/`read_slice`. `write_aggregate_env` and `vb_stages_cmd` both still do a plain `b.write(&w)` over the whole `BlockWitness`. Same disease, 2.4x worse per byte, and never profiled because nothing ever profiled this path. In absolute terms it is 17.9% of the block's entire measured chunk work, paid once per block. It also gets worse toward the tip. Across four blocks the cost per byte rises from 49.3 to 347.2 -- about 7x. It fits `bytes^1.35` overall, but the local exponents are 1.15, 2.34 and 1.04, so no exponent should be quoted from four points on blocks that differ in era as well as size. The direction is what matters and the direction is bad. Two things this contradicts, both recorded rather than resolved. `vb_stages_cmd`'s own comment says the input loop is 73% of the total; it measures 4.5%, so the comment describes a guest that no longer exists. And `reproduce/METHOD_ID` puts the aggregate at ~1,137 M cycles, which is less than half this read alone -- yet the aggregate reads the same witness plus sixteen journals. `HAZYNC_AGG_EXECUTE` settles that, and the sixteen chunk proves are what unblock it, which makes them the highest-value item on the card day: they decide whether a ~2.5 G-cycle inefficiency is real and sitting in the term that becomes 43% of cost after #139.
A finding that reads as "78% of block validation" is not a large lever until it is converted into factor on divisible card-seconds, which is what §1 exists to force. The aggregate is 9.8% of the one-card total, so removing 78% of its validation half is worth 1.06x today -- robust across the 300-500 s uncertainty in what assumption resolution costs -- and 1.30-1.37x once #139 makes the aggregate 43% of the total. So it does not close the 1.92x gap and it changes nothing about §2's conclusion. It is still the best-value item on the board by cost to fix: the technique is already written and shipped in #136, it costs no fidelity, and because the host->guest environment encoding is not guest source it may not move `METHOD_ID` at all -- which no other lever of this size can claim, and which should be checked before anything else about it. It is also worth more later than now, which is the re-rank-forward rule working. A board ranked on today's profile would have put this eighth.
An earlier draft of that section reasoned that the host->guest environment encoding is not guest source, and so might land without a re-baseline. That is wrong. The guest reads it with `let w: BlockWitness = env::read()` at guest main.rs:1186 and :1291, which makes it guest source like any other guest edit. The lever's value is unchanged. Where it sits in the order is not: it batches with Tier 0's codegen and #139 rather than landing on its own, exactly as ACCELERATION.md's "METHOD_ID constraint" section requires.
defenwycke
force-pushed
the
docs/sixteen-card-plan
branch
from
August 27, 2026 00:44
b4d0dfd to
3d71767
Compare
`HAZYNC_WITNESS_SIZES` reports four categories that together account for 34.5% of block 962,000's witness. Measuring the residual across four blocks shows it scales per input at roughly 600-870 bytes, so the `inputs` vector is about 65% of the witness while the transaction bytes everyone would assume dominate are 21%. `BlockInput` is seven scalars plus two `WireProof`s, each carrying a 32-byte leaf, a u64 position and a siblings vector, and sibling counts here are about one per proof -- so the bulk is struct encoding, consistent with hashes and scalars occupying ~4x their raw size in risc0's word stream. This redirects the fix. The obvious reading of the previous section is "pack the transaction bytes", and those are already only a fifth of it. Anyone acting on it should instrument `to_vec` per sub-structure first, because the numbers above are a residual rather than a direct measurement. One more stale comment while here: guest main.rs:898 says the proofs clone ~28x32 B of siblings per input. Measured, it is ~66 B. Do not size anything from it.
§4 called the CUDA layer the loudest unexplained signal on the board and §4.1 noted it is exempt from `METHOD_ID` and fidelity. §8 is that layer, measured on an L40S. **The kernels are nowhere near roofline.** `eval_check` (49.5% of GPU time) runs at 15.2% of peak and `par_stepExec` (19.8%) at 3.7%, both at **255 registers/thread** — one block per SM, ~16% occupancy. H4 holds only for `_poseidon2_rows`, which is 8.1%. **They already spill**: 76-81% of all memory traffic on both is register spill (3.26:1 and 4.17:1 against real data). That invalidates the obvious reading of `eval_check`'s 58.7% of DRAM peak — most of that traffic should not exist, so the ceiling is HIGHER than the 1.70x a roofline reading implies, not lower. The power rail agrees independently: 147 W of 350 W during chunk proving against 276 W during the aggregate, no throttling, same card. And `_poseidon2_rows` is the control that settles blame — same BabyBear arithmetic, same card, **78.3% of peak** at 60 registers. The field arithmetic is not the constraint. **The `>3,300 s` aggregate is stale.** Measured: 117.3 s at N=4, and **1,574.9 s** for the N=16 tip aggregate, GPU never idle. The string predates #148/#153/#157/#161 and is still hardcoded in `report_agg_execute_only`, so every execute run reprints a number measured before its own fix. ⇒ The measurement confirms `FLEET_SIZING.md`'s 1,565.9 s to **0.6%**, so the correction lands on the session notes, not on this plan — §1's 16.6 min stands, and §8.5 now explains where it comes from: the **chunk side** binds, at 980 s for the slowest of sixteen (measured straggler 1.05x against a predicted 1.00x). ⛔ Also downgrades #139 on the chunk side to ~1.1x: bytes do not drive chunk cost (86x the bytes, 4% LESS time) and neither do inputs (3.6x the inputs, 4% more time). Chunks cost EC verifies — Core's libsecp256k1 — which fidelity rightly forbids touching. ⚠ Three things are NOT established and are marked as such in §8.8: whether resolution scales with N (two points, and the block moved between them — a power law through two points fits by construction), whether the distributed aggregate works (claimed by #153/#157/#161, unexercised), and whether the spill is removable at all.
§8.9's first item, run on an L40S: every arm is slower, monotonically. Capping to 64 registers raised occupancy **3.6x** — exactly the mechanism the item hoped for — and **halved throughput**, for +32.3% wall-clock. 128 and 96 have identical occupancy, so 96 bought no warps and merely added 21% more spill. ⇒ Occupancy is not the constraint; spill traffic is. The cap delivers the warps it promises and loses anyway, because it buys them by manufacturing more of the bottleneck. And no flag can fix this: **255 regs/thread is the architectural maximum**, so stock is already at the ceiling and the compiler is spilling by force. The sweep still pays for itself, because it yields a model. Fitting the three capped arms gives **wall ∝ spill^0.21**, so cutting spill 4x buys 1.34x and 32x buys ~2.08x. `_poseidon2_rows` corroborates from the opposite direction: 5.1x the throughput at ~no spill, which the same fit places at a ~32x spill cut. ⇒ The relationship is SUBLINEAR, which re-ranks the remaining work. Halving spill buys 16%. Only splitting `eval_check` into passes (item 3) has the shape to nearly eliminate it, and its traffic budget is favourable: 7.16 G global against 23.36 G spill today, so 4 passes is roughly flat on total sectors while converting scattered local into coalesced global. Control: vendored `stock` arm reproduced the pre-vendoring binary to 0.2% on wall-clock and to the sector on spill counters; `METHOD_ID` identical on all four arms.
…arrangement
Five arms on the same segment, median of 9, `METHOD_ID` identical throughout. Every one is
slower than stock:
stock 255 regs 1 blk/SM 15.21% 23.36 G local 4,115 ms
__noinline__ 255 1 14.85% 23.85 4,154 ms +0.9%
-dlto 255 1 22.24% 45.72 4,260 ms +3.5%
-maxrreg 128 128 2 11.74% 40.89 4,552 ms +10.6%
-maxrreg 64 64 4 7.78% 67.13 5,446 ms +32.3%
`-dlto` settles it. LTO genuinely inlined across the translation units — throughput rose
**46%**, which nothing else explains — and local traffic still **doubled**. It removed
`poly_fp`'s 6,952-byte ABI frame and the twenty per-call frames, and the union live set then
overflowed 255 registers by more than the frames had cost.
⇒ The live set does not fit under ANY arrangement. Registers pin at 255 — the architectural
maximum — in all five arms. The compiler offers exactly three degrees of freedom here: call
it (ABI frames), inline it (register spill), cap it (more register spill). All three are now
measured, and nvcc's default is the cheapest.
⇒ The 1.3-2.1x kernel lever is not reachable by any means available to this project. Getting
it needs the constraint polynomial restructured in RISC0's circuit generator — upstream, in
a repo dormant on two filed issues. The board returns to fan-out (§8.6: chunk cost tracks EC
verifies, which divide by N), which makes §8.8's resolution-scaling question the highest
-value open item.
⚠ The `unity` arm was staged and deliberately NOT run — it reaches the same mechanism as
`-dlto` via the compiler rather than the linker, and would have re-measured a settled
question.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The board carries many levers quoted in units that do not compose — execute-mode cycles, one-card wall-clock, per-chunk factors, block-level bounds — so nobody can say whether any of them reaches a stated target. This adds
docs/SIXTEEN_CARD_PLAN.md: the target (a near-tip block on 16 L40S), the single denominator every lever must be priced in, and the arithmetic that follows from it.The arithmetic
All known zero-fidelity levers, multiplied, move a block from 18.2 min to 16.6 min against a 10.0 min target. Tier 0 codegen is
1.012xMEASURED; the efficiency axis is≤1.09x(below). That is the whole remaining board, and it closes about a fifth of the gap in seconds and none of it in fidelity terms.Two corrections, from E7's evidence rather than from argument
hazync-l40s4measure 91.5% and 91.7% mean utilisation,nvidia-smisampled once per second, with the only zeros in the first 8 s — execute, before proving starts.PERF_INVESTIGATION_2026-08-26.md§4.2 asserts 65% idle and §1 of the same file asserts 65% busy; the vmstat evidence behind the former shows the host is single-threaded and says nothing about the card. E6's ceiling drops to~1.09x, and E9 should close — it proposes a fork of a dormant upstream, carrying a CUDA deadlock we have already hit in CUDA prove-chunk hung for 76 minutes with the GPU idle and produced no output at all #147/Make proving divide: segment distribution and a balanced join tree #148, to chase1.09x.nvidia-smireports kernel residency, not efficiency. So the above kills the idle-gap framing and says nothing about what the card is doing while it is busy. That question has never been asked here:grep -riE "nsight|ncu|nsys|occupancy"returns zero hits across the repo.Which reframes the loudest signal on the board
Three architectures land within 9% of each other across a 4x difference in memory bandwidth (H100
0.95x, B2000.91x).FLEET_SIZING.mdreads that as "the card axis is closed", which is the right operational conclusion and the wrong diagnostic one. Performance invariant to bandwidth and to two architecture generations is not a fact about cards — it is a fact about our kernels. And kernel work moves neitherMETHOD_IDnor fidelity, in arisc0-syswe already vendor (#182) against an upstream §E10 establishes is dormant.The method
A four-layer profile — guest cycles, risc0 phases, host, device — that must reconcile to the same 399 seconds. If the layers do not sum to the same wall, the profile is wrong and no lever derived from it is trustworthy. L1 is done and exhausted at 1.16%; L2, L3 and L4 have never been run. §5.1 gives the four hypotheses one
ncurun separates, and each is decision-relevant: three of them open a zero-fidelity lever class, and the fourth closes the efficiency axis and settles the #139 argument by elimination.Plus a rule that would have caught two known misses: rank levers against the profile that will exist after the others land. The packer refit — worth
2.36x, a prerequisite for #139 rather than a follow-up — was found late for exactly this reason, and the aggregate is 9.6% of cost today and 43% after #139, still unprofiled, withHAZYNC_AGG_EXECUTE=1sitting there costing nothing to run.Also priced here for the first time
prove_chunktakes no previous receipt — the chain dependency isadd_assumption(prev)inprove_step(main.rs:396). So only the per-block fold is sequential, and blocks may be proved overlapped at a bounded lag, which overlaps consecutive serial floors and drops the bar from1.95xto1.69xfor nothing.Docs only.
scripts/check-versions.shpasses.