From baeb90d7b83670a26a52f4a4a618552a00333e92 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 22:08:13 +0000 Subject: [PATCH 1/8] board: record merged #946 (arc entry + shipped row) #946 was MIXED -- probe code, real measurements, two epiphanies and a synthesis document -- so it owes both obligations of the merged-PR row. The arc entry carries what a future session would otherwise re-derive: that D-CZ-0 was marked DONE with no artifact behind it and that #945's own audit compared prose to prose; the gradient definition identified FROM DATA as Pa-per-cell-without-cos(lat) and its bounded consequence (order survives, magnitudes do not); D-CZ-1's gate passing 19/19 after the sample-composition fix; the C4 amendment WITH its legitimacy scope (valid only while no C4 cell has been scored); the disable-verified selftest including the tie-averaging break that flips a sign silently; and the exploratory correlation recorded with its not-a-result status string so a later run cannot restate it as a confirmation. Also records the formula matrix: 46 rated primitives, 20 D and 3 V, i.e. half the inventory is a negative result -- with the C tier explained (a comfort zone is a map, not a grade) and the D/V distinction stated (D lost its test; V distinguished nothing). Confidence split recorded honestly: [G] on the gate, the reproduction, the definition identification and the rho-saturation measurement; [H] on the C4 amendment's scope; and the two exploratory items explicitly NOT results. PR_ARC_INVENTORY prepend suffix-verified. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/LATEST_STATE.md | 1 + .claude/board/PR_ARC_INVENTORY.md | 76 +++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index d1b1fa7d4..69ffcf40e 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -855,6 +855,7 @@ Membrane consumers can now pull BOTH halves of a render `classid` BBB-safely fro | PR | Merged | Title | What it added | |---|---|---|---| | *gap note* | — | **#781–#925 are NOT in this table** — carried by the dated sections above + `PR_ARC_INVENTORY.md`. Recorded 2026-08-12 (codex P2 on #930) rather than silently reconstructed; the table had stalled at #780. | — | +| **#946** | 2026-08-12 | D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix | MIXED (probe code + measurements + 2 epiphanies + a synthesis doc). **D-CZ-0 was marked DONE with no script and no JSON** — quoted in the plan, the arc, the LATEST_STATE row and 3 PR bodies; #945's self-audit called it "verified" while only comparing arc entry to PLAN (prose vs prose). Reproduced 4/9 rows (1.004/1.022/0.994/0.931); the 5 excluded land candidates are **unreproducible** (centres never recorded; no coordinates invented). `|∇p|` definition identified FROM DATA as **Pa/cell without cos(lat)** (max dev 0.069 vs next-best 0.398) → R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, **ORDER survives**, range 9.3× → **9.8×**. **D-CZ-1 PASS**: both controls lose on both metrics in all 4 regimes and in **19/19 storms**; `GEO-DEGENERATE` saturates 92–97 %; **C1b separation 6.28** vs ≥3. **The run AMENDED C4**: ρ saturated on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps ρ off-diagonal, **C4 moves to RMSE in Pa**; amendment carries its trigger and was propagated to header + bar. Committed **`--selftest`**, each assertion disable-verified (dropping tie-averaging **flips the sign** of heavy-ties ρ). Exploratory within-R4 ρ=+0.444, n=19, **p=0.0578 (not significant)** — and the tautology check does NOT dismiss it (confounds −0.035/+0.253/−0.081); committed WITH a not-a-result status string. **Plus `SUBSTRATE_FORMULA_MATRIX.md`**: 46 rated primitives (15 A / 4 B / 4 C / **20 D / 3 V** — half the inventory is negative), 14 known-vs-discovered pairs, 9 apparatus lessons, 13 gaps; built by re-extracting from committed artifacts (131 primitives, 2.35 M subagent tokens), **28 figures re-verified, 0 mismatches, 1 rounding fixed**. Every off-diagonal cross-swap cell remains unmeasured. | | **#944** | 2026-08-12 | `substrate-comfort-zones-v1` §2 rebuilt: horse race → CROSS-SWAP transfer matrix | MIXED (plan revision + epiphany + a review-caught correction). Operator ruling: the method is cross-swap under the premise the model CAPTURES the phenomenon but is NOT calibrated — so miscalibration is the measurement CONDITION, not an arm, and RMSE under a deliberately wrong calibration is bad BY DEFINITION. §2 is now a **4×4 donor × target matrix** `M[D][T]`; primary metric Spearman `ρ`; the hypothesis's quantity is the derived **transfer loss** `L[D][T] = ρ[T][T] − ρ[D][T]`. `CAL-ABS-OWN`/`CAL-ABS-FOREIGN` collapse into ONE arm's diagonal/off-diagonal. The dynamic arms have **no donor**, so `L ≡ 0` by construction — the property under test, and **C2 requires BOTH** that it hold exactly AND that `CAL-ABS` be proven non-degenerate through the same code path (the can-it-DIFFER gate W5 paid for). Two NEW bars for the operator's constancy-is-relative point: **C1b** MEASURES constancy (`separation ≥ 3`) instead of claiming it; **C1c** gives the suitability ASSUMPTION a falsifier — regimes must differ in autocorrelation decay + rank-distribution shape, not merely in `\|∇p\|`, and a null there VOIDS the reading. `L < 0` pre-registered as informative (a donor's wider range can legitimately beat an outlier-set diagonal). **CodeRabbit caught the sticky half**: §2.3 still named the FOREIGN comparison as the hypothesis while C4/board scored against the diagonal — two incompatible verdict criteria, the weaker holding the headline; fixed `1756c518`. Fifth instance this arc of a claim consistent with its own operands but inconsistent with a SIBLING claim. Unchanged: the 4-tier regime ladder + its three preflight corrections, equal budget, geometry axis, C0 gate, output contract. **D-CZ-1 (control losability) still Queued and still gates every later cell.** Zero code, zero probe runs, zero fetches. | | **#942** | 2026-08-12 | "mean" qualifier fix + `substrate-comfort-zones-v1` (calibration × regime, made falsifiable) | MIXED. Plan states the operator's hypothesis as an INTERACTION (a poorly-calibrated adaptive substrate does relatively better in strong/turbulent storms than in calm regimes) across **4** held-constant regimes × formula × calibration quality — the preflight SPLIT "flatland" into calm and active tiers: R1 CALM Amazon (`\|∇p\|` 10.2) → R2 OCEAN S-Pacific (14.9) → R3 ACTIVE W-Siberia (43.8) → R4 STORM (95.6), a 9.3× range. "Worse everywhere but least-worse in storms" ≠ "better only in storms", and the plan distinguishes them. D-CZ-0 preflight DONE (sample-composition arithmetic run before the first fetch — it is what forced the 4-way split; units annotated at coefficient definition). **D-CZ-1, the control-losability smoke test, is PRESCRIBED BY the plan but Queued — not yet run**, and gates every later cell. D-CZ-2..6 Queued. Also fixed CodeRabbit's Minor on #941 — #940's 25 %/9 % figures are MEANS over 19 storms, qualifier restored (4th instance this week of scope-lost-in-summary). | | **#940** | 2026-08-12 | W6 lands — vector-sum dipole model VOID by its own anti-vacuity control; stranded stratum empty by CT-F14's filter arithmetic; same-PR sign/units correction round | B0 VOID: single-geo R²=−0.104 (worse than the mean); both controls (permuted P_bow, P_bow rotated 90°) clear the `≤single-geo+0.03` ceiling. B3 stranded (`\|v_storm\|<8 m/s`): n=0 — `displacement_km≥250`/6h implies `\|v_storm\|≥11.57 m/s` for every admitted storm, `E-THE-DISPLACEMENT-FILTER-ATE-THE-STRANDED-STRATUM-1`. Same-PR: codex+CodeRabbit caught a sign-convention bug (`D=-spine(...)`, matching `low_pole_bearing()`'s own flip; verified offline that this leaves R²/B0/B1/B3 unchanged, confirmed on the actual re-run) and a units error (`c_bow` is km⁻¹ not dimensionless; replaced with the dimensionally valid `\|c_bow·P_bow\|` vs `\|D\|` metric — mean-over-19-storms: geo≈25%, bow≈9% of mean `\|D\|`). CT-F17's gate now moot for this model form. | diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 341c004ee..b30301e79 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,79 @@ +## 2026-08-12 — lance-graph #946 (MERGED) — D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix + +- **The finding that forced the run.** `substrate-comfort-zones-v1` §1's + nine-row regime preflight was marked **DONE** on `STATUS_BOARD`, quoted in + the arc entry, quoted again in the `LATEST_STATE` row, and its 9.3× headline + carried into three PR bodies — **with no script and no JSON producing any of + it.** Worse: #945's self-audit, run *specifically because* this arc keeps + shipping unbacked summary claims, reported those figures as "verified" while + only comparing the arc entry to the PLAN. Both are prose. **A figure cited in + two documents is cited twice, not confirmed once**, and an audit that compares + two documents measures their consistency — not the property anyone cares about. +- **Reproduced — 4 of 9 rows** (ratios 1.004 / 1.022 / 0.994 / 0.931). The five + EXCLUDED land candidates are **unreproducible**: their box centres were never + written down anywhere, and **no coordinates were invented** to fake them. +- **The `|∇p|` definition was never committed either** — so four candidates were + computed and the winner decided **from the data**: Pa per grid cell with **NO + cos(lat) metric** (max dev 0.069; next-best 0.398), a plain `np.gradient` over + the raw array. It understates the zonal gradient by `1/cos(lat)`, so **R3 at + 60 N is ~40 % low**. Metric-corrected the ladder is 10.3 / 15.5 / 61.2 / 100.9 + — **ORDER survives**, range widens 9.3× → **≈9.8×**. Regime axis stands; the + recorded magnitudes do not. +- **Two defects in the reproduction, both caught by its own guards.** Measuring + all 19 storms at the *preflight* timestep instead of each storm's own `t0` + (inverted R3/R4 and would have read as C1 failing); and a seam assertion firing + on a real storm at 353.4 E, where a plain longitude slice yields 58 columns + instead of 65. +- **D-CZ-1 PASSES.** Both controls lose to both real arms, on both metrics, in + all four regimes — and, after the sample-composition fix, in **19/19 storms**, + not merely at the median. `GEO-DEGENERATE` saturates **92–97 %**, so the + mechanism is visible rather than assumed. **C1b `separation` = 6.28** vs ≥ 3. +- **…and the run AMENDED C4.** `ρ` is **saturated on the diagonal** (real-arm + spread **3×10⁻⁶ … 4.7×10⁻⁵**) — C4 as pre-registered **could not have fired**. + `L` keeps `ρ` off-diagonal (four orders of range); **C4 moves to RMSE in Pa** + with `ρ` as a floor check. Legitimate because D-CZ-1's purpose is to test the + apparatus before the expensive cells and **no C4 cell has been scored**; + illegitimate the instant one has. Amendment carries its trigger and was + **propagated** to the header and the bar — the propagation failure a reviewer + caught on #944. +- **Committed `--selftest`, disable-verified.** ρ vs scipy incl. heavy ties, + exact ±1 anchors, `nan` on constant input; saturation 0.802 vs 0.008; + `rank_codec` monotone over 256 levels; seam box keeps 65 distinct columns. + Each assertion **broken on purpose** to confirm it can fail — dropping + tie-averaging **flips the sign** of the heavy-ties ρ (+0.054 vs −0.031), which + would have corrupted silently rather than crashing. +- **Exploratory, and it survives its own dismissal.** Within R4: + ρ(`GEO-DEGENERATE` ρ, storm `|∇p|`) = **+0.444**, n=19, permutation + **p = 0.0578** (200 000 perms) — **above 0.05, not significant**. The + tautology check does NOT hold (ρ vs box range −0.035; `|∇p|` vs box range + +0.253; ρ vs donor/box fraction −0.081), so neither confirmation nor easy + dismissal is available. Committed as `exploratory_within_r4` **with a status + string saying it is not a result**, so a later run cannot restate it as a + confirmation that was there all along. +- **Added: `probes/weather-p1/SUBSTRATE_FORMULA_MATRIX.md`** — the arc's rated + inventory. Built by re-extracting from the COMMITTED artifacts (7 parallel + readers + 1; **131 primitives, 2.35 M subagent tokens**), never from session + memory. **Two scales, deliberately unmerged:** fitness (A/B/C/D/V) × evidence + (`[G]`/`[H]`/`[S]`). **46 rated: 15 A, 4 B, 4 C, 20 D, 3 V — half the + inventory is a negative result.** A **C is not a bad grade** (Fisher-z alone + carries three verdicts: tail 8.3× win, level 4.7× loss, CI-frame not-a-win); + **D and V differ in kind** (D lost its test; V distinguished nothing). + 14 known-vs-discovered pairs, four recording the measurement going AGAINST the + prior. Figures verified against source JSONs: **28 checked, 0 mismatches, 1 + rounding fixed** (Rayleigh p 0.689 → 0.688). +- **Locked.** The regime ladder's ORDER; the C0 control gate as a real gate; + `ρ` for transfer loss; RMSE in Pa for the crossover. +- **Deferred.** D-CZ-2, 2b, 2c, 3..7 — **every off-diagonal cross-swap cell is + still unmeasured**. D-MTX-5 (refresh the matrix after they run). +- **Docs.** `EPIPHANIES.md` `E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1` + + `E-THE-METRIC-THAT-SEPARATES-ONE-COMPARISON-IS-BLIND-TO-ANOTHER-1`; + `STATUS_BOARD` D-CZ-0/D-CZ-1 updated, D-MTX-1..5 added; plan §6 RUN section. +- **Confidence.** `[G]` on D-CZ-1's gate, the reproduction, the definition + identification, and the ρ-saturation measurement — all committed and + re-runnable. `[H]` on the C4 amendment's *scope* (correct here, illegitimate + once a C4 cell exists). **Explicitly NOT a result:** the §6.5 RMSE-margin hint + and the §6.6 exploratory correlation, both labelled as such in the artifact. + ## 2026-08-12 — lance-graph #944 (MERGED) — `substrate-comfort-zones-v1` §2 rebuilt: horse race → CROSS-SWAP transfer matrix - **Added.** A rebuilt §2 for `.claude/plans/substrate-comfort-zones-v1.md` From 0fb8431ccf5e4c87610d907fc529c379c17e9b4b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 22:10:43 +0000 Subject: [PATCH 2/8] board: correct a stale saturation figure my own audit caught (92-97% -> 72-97%) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-audited the #946 arc entry against the MERGED artifacts rather than against the PR body -- 17 checkable claims, 16 verified, 1 wrong. "GEO-DEGENERATE saturates 92-97%" was TRUE when first written (R1 0.9496 / R2 0.9174 / R3 0.9718 / R4 0.9179) and went stale INSIDE THE SAME PR: fixing R4 to measure each storm at its own t0 moved R4 to 0.7224. Committed reality is 72-97% per regime; across all 19 storms min 0.693 / median 0.843 / max 0.974. I carried the stale range into four files and two PR bodies. The verdict is unaffected -- 72% is still overwhelming saturation and the gate passes 19/19 -- but the number was wrong, and this is the seventh instance in this session of the same defect class: a figure true when written, stale once the artifact beneath it changed, carried forward because the check compared prose to prose. It is the first one my own audit caught rather than a reviewer, and only because the audit terminated at the JSON. Fixed in the unmerged #947 files (arc entry, LATEST_STATE) in place; for the MERGED plan the correction is APPENDED as a dated note rather than silently edited, and STATUS_BOARD's row carries the correction inline. And the corrected number is the more interesting one: the STORM regime saturates LEAST (0.72 vs 0.92-0.97 in the calmer tiers) -- the degenerate donor hurts less where the field is strongest, which is independently the direction the §6.6 exploratory correlation measured (rho = +0.444). Two measurements that were never connected agree. Recorded as a coherence, NOT as evidence: §6.6 is still p = 0.0578, above 0.05. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/LATEST_STATE.md | 2 +- .claude/board/PR_ARC_INVENTORY.md | 2 +- .claude/board/STATUS_BOARD.md | 2 +- .claude/plans/substrate-comfort-zones-v1.md | 24 ++++++++++++++++++++- 4 files changed, 26 insertions(+), 4 deletions(-) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 69ffcf40e..2c7424e64 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -855,7 +855,7 @@ Membrane consumers can now pull BOTH halves of a render `classid` BBB-safely fro | PR | Merged | Title | What it added | |---|---|---|---| | *gap note* | — | **#781–#925 are NOT in this table** — carried by the dated sections above + `PR_ARC_INVENTORY.md`. Recorded 2026-08-12 (codex P2 on #930) rather than silently reconstructed; the table had stalled at #780. | — | -| **#946** | 2026-08-12 | D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix | MIXED (probe code + measurements + 2 epiphanies + a synthesis doc). **D-CZ-0 was marked DONE with no script and no JSON** — quoted in the plan, the arc, the LATEST_STATE row and 3 PR bodies; #945's self-audit called it "verified" while only comparing arc entry to PLAN (prose vs prose). Reproduced 4/9 rows (1.004/1.022/0.994/0.931); the 5 excluded land candidates are **unreproducible** (centres never recorded; no coordinates invented). `|∇p|` definition identified FROM DATA as **Pa/cell without cos(lat)** (max dev 0.069 vs next-best 0.398) → R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, **ORDER survives**, range 9.3× → **9.8×**. **D-CZ-1 PASS**: both controls lose on both metrics in all 4 regimes and in **19/19 storms**; `GEO-DEGENERATE` saturates 92–97 %; **C1b separation 6.28** vs ≥3. **The run AMENDED C4**: ρ saturated on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps ρ off-diagonal, **C4 moves to RMSE in Pa**; amendment carries its trigger and was propagated to header + bar. Committed **`--selftest`**, each assertion disable-verified (dropping tie-averaging **flips the sign** of heavy-ties ρ). Exploratory within-R4 ρ=+0.444, n=19, **p=0.0578 (not significant)** — and the tautology check does NOT dismiss it (confounds −0.035/+0.253/−0.081); committed WITH a not-a-result status string. **Plus `SUBSTRATE_FORMULA_MATRIX.md`**: 46 rated primitives (15 A / 4 B / 4 C / **20 D / 3 V** — half the inventory is negative), 14 known-vs-discovered pairs, 9 apparatus lessons, 13 gaps; built by re-extracting from committed artifacts (131 primitives, 2.35 M subagent tokens), **28 figures re-verified, 0 mismatches, 1 rounding fixed**. Every off-diagonal cross-swap cell remains unmeasured. | +| **#946** | 2026-08-12 | D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix | MIXED (probe code + measurements + 2 epiphanies + a synthesis doc). **D-CZ-0 was marked DONE with no script and no JSON** — quoted in the plan, the arc, the LATEST_STATE row and 3 PR bodies; #945's self-audit called it "verified" while only comparing arc entry to PLAN (prose vs prose). Reproduced 4/9 rows (1.004/1.022/0.994/0.931); the 5 excluded land candidates are **unreproducible** (centres never recorded; no coordinates invented). `|∇p|` definition identified FROM DATA as **Pa/cell without cos(lat)** (max dev 0.069 vs next-best 0.398) → R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, **ORDER survives**, range 9.3× → **9.8×**. **D-CZ-1 PASS**: both controls lose on both metrics in all 4 regimes and in **19/19 storms**; `GEO-DEGENERATE` saturates 72–97 %; **C1b separation 6.28** vs ≥3. **The run AMENDED C4**: ρ saturated on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps ρ off-diagonal, **C4 moves to RMSE in Pa**; amendment carries its trigger and was propagated to header + bar. Committed **`--selftest`**, each assertion disable-verified (dropping tie-averaging **flips the sign** of heavy-ties ρ). Exploratory within-R4 ρ=+0.444, n=19, **p=0.0578 (not significant)** — and the tautology check does NOT dismiss it (confounds −0.035/+0.253/−0.081); committed WITH a not-a-result status string. **Plus `SUBSTRATE_FORMULA_MATRIX.md`**: 46 rated primitives (15 A / 4 B / 4 C / **20 D / 3 V** — half the inventory is negative), 14 known-vs-discovered pairs, 9 apparatus lessons, 13 gaps; built by re-extracting from committed artifacts (131 primitives, 2.35 M subagent tokens), **28 figures re-verified, 0 mismatches, 1 rounding fixed**. Every off-diagonal cross-swap cell remains unmeasured. | | **#944** | 2026-08-12 | `substrate-comfort-zones-v1` §2 rebuilt: horse race → CROSS-SWAP transfer matrix | MIXED (plan revision + epiphany + a review-caught correction). Operator ruling: the method is cross-swap under the premise the model CAPTURES the phenomenon but is NOT calibrated — so miscalibration is the measurement CONDITION, not an arm, and RMSE under a deliberately wrong calibration is bad BY DEFINITION. §2 is now a **4×4 donor × target matrix** `M[D][T]`; primary metric Spearman `ρ`; the hypothesis's quantity is the derived **transfer loss** `L[D][T] = ρ[T][T] − ρ[D][T]`. `CAL-ABS-OWN`/`CAL-ABS-FOREIGN` collapse into ONE arm's diagonal/off-diagonal. The dynamic arms have **no donor**, so `L ≡ 0` by construction — the property under test, and **C2 requires BOTH** that it hold exactly AND that `CAL-ABS` be proven non-degenerate through the same code path (the can-it-DIFFER gate W5 paid for). Two NEW bars for the operator's constancy-is-relative point: **C1b** MEASURES constancy (`separation ≥ 3`) instead of claiming it; **C1c** gives the suitability ASSUMPTION a falsifier — regimes must differ in autocorrelation decay + rank-distribution shape, not merely in `\|∇p\|`, and a null there VOIDS the reading. `L < 0` pre-registered as informative (a donor's wider range can legitimately beat an outlier-set diagonal). **CodeRabbit caught the sticky half**: §2.3 still named the FOREIGN comparison as the hypothesis while C4/board scored against the diagonal — two incompatible verdict criteria, the weaker holding the headline; fixed `1756c518`. Fifth instance this arc of a claim consistent with its own operands but inconsistent with a SIBLING claim. Unchanged: the 4-tier regime ladder + its three preflight corrections, equal budget, geometry axis, C0 gate, output contract. **D-CZ-1 (control losability) still Queued and still gates every later cell.** Zero code, zero probe runs, zero fetches. | | **#942** | 2026-08-12 | "mean" qualifier fix + `substrate-comfort-zones-v1` (calibration × regime, made falsifiable) | MIXED. Plan states the operator's hypothesis as an INTERACTION (a poorly-calibrated adaptive substrate does relatively better in strong/turbulent storms than in calm regimes) across **4** held-constant regimes × formula × calibration quality — the preflight SPLIT "flatland" into calm and active tiers: R1 CALM Amazon (`\|∇p\|` 10.2) → R2 OCEAN S-Pacific (14.9) → R3 ACTIVE W-Siberia (43.8) → R4 STORM (95.6), a 9.3× range. "Worse everywhere but least-worse in storms" ≠ "better only in storms", and the plan distinguishes them. D-CZ-0 preflight DONE (sample-composition arithmetic run before the first fetch — it is what forced the 4-way split; units annotated at coefficient definition). **D-CZ-1, the control-losability smoke test, is PRESCRIBED BY the plan but Queued — not yet run**, and gates every later cell. D-CZ-2..6 Queued. Also fixed CodeRabbit's Minor on #941 — #940's 25 %/9 % figures are MEANS over 19 storms, qualifier restored (4th instance this week of scope-lost-in-summary). | | **#940** | 2026-08-12 | W6 lands — vector-sum dipole model VOID by its own anti-vacuity control; stranded stratum empty by CT-F14's filter arithmetic; same-PR sign/units correction round | B0 VOID: single-geo R²=−0.104 (worse than the mean); both controls (permuted P_bow, P_bow rotated 90°) clear the `≤single-geo+0.03` ceiling. B3 stranded (`\|v_storm\|<8 m/s`): n=0 — `displacement_km≥250`/6h implies `\|v_storm\|≥11.57 m/s` for every admitted storm, `E-THE-DISPLACEMENT-FILTER-ATE-THE-STRANDED-STRATUM-1`. Same-PR: codex+CodeRabbit caught a sign-convention bug (`D=-spine(...)`, matching `low_pole_bearing()`'s own flip; verified offline that this leaves R²/B0/B1/B3 unchanged, confirmed on the actual re-run) and a units error (`c_bow` is km⁻¹ not dimensionless; replaced with the dimensionally valid `\|c_bow·P_bow\|` vs `\|D\|` metric — mean-over-19-storms: geo≈25%, bow≈9% of mean `\|D\|`). CT-F17's gate now moot for this model form. | diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index b30301e79..ae6035f2c 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -26,7 +26,7 @@ instead of 65. - **D-CZ-1 PASSES.** Both controls lose to both real arms, on both metrics, in all four regimes — and, after the sample-composition fix, in **19/19 storms**, - not merely at the median. `GEO-DEGENERATE` saturates **92–97 %**, so the + not merely at the median. `GEO-DEGENERATE` saturates **72–97 %**, so the mechanism is visible rather than assumed. **C1b `separation` = 6.28** vs ≥ 3. - **…and the run AMENDED C4.** `ρ` is **saturated on the diagonal** (real-arm spread **3×10⁻⁶ … 4.7×10⁻⁵**) — C4 as pre-registered **could not have fired**. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index e91a73b4b..86b61340f 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -34,7 +34,7 @@ already run and it corrected two regime definitions before any bar existed. | D-id | Deliverable | Status | Feeds | |---|---|---|---| | D-CZ-0 | §1 regime preflight (`\|∇p\|` ladder, elevation-confound screen, speed-is-not-the-discriminator finding) | **DONE — and now REPRODUCED + partly corrected (§6.1/6.2).** It had NO committed script or JSON when marked DONE. 4 rows reproduce (1.004/1.022/0.994/0.931); the 5 EXCLUDED land candidates are **unreproducible** (centres never recorded). Definition identified: Pa per grid cell **without cos(lat)** — R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, order survives, range 9.3× → **9.8×**. Original text:** — ladder R1 Amazon 10.2 → R2 ocean 14.9 → R3 W Siberia 43.8 → R4 storm 95.6 (9.3× range); 4 land candidates excluded on elev σ > 150 m | the regime axis all other rows score on | -| D-CZ-1 | C0 controls (shuffled codebook + degenerate geometry), losability-smoke-tested BEFORE the full run | **DONE — PASS** (`substrate_comfort_d_cz_0_1.py/.json`). Both controls lose to both real arms on BOTH metrics in ALL FOUR regimes; mechanism visible (`GEO-DEGENERATE` saturates 92–97 %). C1b `separation` = **6.28** vs the ≥ 3 bar. **AND the run amended C4**: `ρ` is SATURATED on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps `ρ` off-diagonal, C4 moves to RMSE in Pa (§6.4) | gates every cell — a control that can't lose voids its cell | +| D-CZ-1 | C0 controls (shuffled codebook + degenerate geometry), losability-smoke-tested BEFORE the full run | **DONE — PASS** (`substrate_comfort_d_cz_0_1.py/.json`). Both controls lose to both real arms on BOTH metrics in ALL FOUR regimes; mechanism visible (`GEO-DEGENERATE` saturates **72–97 %** per regime — corrected 2026-08-12 from a stale 92–97 %, see plan §6.3; R4 is the low end at 0.7224). C1b `separation` = **6.28** vs the ≥ 3 bar. **AND the run amended C4**: `ρ` is SATURATED on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps `ρ` off-diagonal, C4 moves to RMSE in Pa (§6.4) | gates every cell — a control that can't lose voids its cell | | D-CZ-2 | C1 regime-ladder stability across ≥3 timesteps | Queued | anti-cherry-pick on the whole regime axis | | D-CZ-2b | **C1b constancy is measured** — `separation = (between-box range)/(mean within-box σ)` ≥ 3 | Queued | earns the phrase "held constant"; otherwise a caveat rides every cell | | D-CZ-2c | **C1c the suitability ASSUMPTION** — regimes must differ in autocorrelation decay + rank-distribution shape, not only in `\|∇p\|` | Queued | a null here VOIDS the cross-swap reading (spread manufactured on the wrong axis) | diff --git a/.claude/plans/substrate-comfort-zones-v1.md b/.claude/plans/substrate-comfort-zones-v1.md index 5b1b51202..592402f03 100644 --- a/.claude/plans/substrate-comfort-zones-v1.md +++ b/.claude/plans/substrate-comfort-zones-v1.md @@ -487,10 +487,32 @@ regime axis stands; the recorded magnitudes do not. Both controls lose to both real arms, on both metrics, in **all four** regimes. The mechanism is visible rather than assumed: `GEO-DEGENERATE` -saturates **92–97 %** of the box because its donor patch's range is far too +saturates **72–97 %** of the box because its donor patch's range is far too narrow — which is exactly the failure mode the transfer matrix is built to measure. +> **⚠ FIGURE CORRECTED (2026-08-12, self-audit on #947).** This read +> **92–97 %** when first written, and that was TRUE of the numbers then on +> disk (R1 0.9496 / R2 0.9174 / R3 0.9718 / R4 0.9179). It went stale in +> the SAME PR: fixing R4 to measure each storm at its own `t0` moved R4's +> saturation to **0.7224**, and the range with it. Per-regime as committed: +> **R1 0.9496 · R2 0.9174 · R3 0.9718 · R4 0.7224**; across all 19 storms +> min **0.693** / median **0.843** / max **0.974**. +> +> **The verdict is unaffected** — 72 % is still overwhelming saturation and +> the gate passes 19/19 — but the figure was wrong in four files and two PR +> bodies. Seventh instance of the arc's recurring defect: a number true when +> written, stale once the artifact beneath it changed, and carried forward +> because the check compared prose to prose. Caught only by re-verifying +> against the JSON. +> +> **And the corrected number is more interesting than the wrong one.** The +> storm regime saturates *least* (0.72 vs 0.92–0.97 in the calmer tiers) — +> i.e. the degenerate donor hurts **less** where the field is strongest, +> which is independently the direction §6.6's exploratory correlation +> measured (ρ = +0.444). Two measurements that were never connected agree. +> Recorded as a coherence, NOT as evidence: §6.6 is still p = 0.0578. + | regime | CAL-ABS ρ | CAL-RANK ρ | SHUFFLE ρ | DEGENERATE ρ (satur.) | |---|---|---|---|---| | R1 | 0.999960 | 0.999993 | 0.058 | 0.380 (0.950) | From ac3bcf45bbe7e6a7de7e57a61b91f1c26da306f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 22:29:45 +0000 Subject: [PATCH 3/8] D-CZ-2..7 RUN: the cross-swap hypothesis is refuted, cleanly and reversed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the pending bars from substrate-comfort-zones-v1.md §3, in the order the plan requires: C1c first, because a null there voids the interpretation before anything downstream is worth trusting. C0 GATE CAUGHT A REAL BUG on its own first use beyond a smoke test. GEO-DEGENERATE was built from truth[:len//64] -- the first slice of an array already shuffled by rng.choice for equal budget. Shuffled means no spatial order, so that slice is an ordinary random subsample, not a narrow spatial patch -- a materially different construction from D-CZ-1's correct one (p[si,sj][:n_i,:n_i], a genuine 2-D corner). Fixed by carrying each regime's full 2-D box alongside the flat evaluation sample and building the degenerate donor from a real spatial corner, matching D-CZ-1 exactly. Disable-verified: reverting to the flat-slice construction reproduces the EXACT original failure (R1 and R4 fail C0); the fix reproduces the IDENTICAL real-arm numbers (C2-C6 unchanged) while making C0 pass cleanly in all four regimes. The bug lived entirely in the control. C1c PASSES and licenses the whole exercise: decay length / Gini / tail ratio of |grad p|, R4-vs-R1 ratios 0.88 / 0.71 / 0.385 -- all deviate >=20% from 1, the regimes differ in correlation STRUCTURE, not merely in gradient magnitude. C1 (ladder) and C1b (separation 5.87-8.24) both PASS across 3 independent timesteps. C2 PASSES both halves: dynamic arms show L==0 exactly for every donor; CAL-ABS is demonstrably non-degenerate through the identical code path (max|L| ranges 0.011 to 0.947). C3 FAILS, and reversed: L-bar = 0.011 -> 0.309 -> 0.671 -> 0.690, R1->R4 -- a MONOTONIC INCREASE, the opposite of the required decrease. Storms are LESS forgiving of bad calibration on this data, not more. C4 FAILS, same direction: absolute wins its own diagonal in ALL FOUR regimes on RMSE, margin GROWING from ~1 Pa to 10.78 Pa in the storm regime -- no crossover, no flip, the opposite of a crossover. The weak form passes trivially as pre-registered (329-2508 Pa, "evidence of wiring, not merit") and is reported separately, never conflated with C4's real bar. C5 NOT RUN: structural blocker, not a gap. The golden index floor needs N >= F(17)^2 = 2,550,409; a 16-degree box holds 4225 cells, three orders of magnitude short. GEO-GOLDEN-HI has no admissible construction at box scale on this grid. Reported, not faked with an interpolated lattice. C6 delivered: full donor x target matrix, all 5 arms x 4 regimes, every cell raw. This directly CONTRADICTS §6.6's earlier exploratory hint (rho=+0.444, p=0.058, direction matching the hypothesis, explicitly labelled not a result). Stated plainly rather than averaged away: the pre-registered cross-regime test governs where the two disagree. Two new instruments (gini, decay_length_cells) plus tail_ratio verified against synthetic ground truth before trusting C1c's numbers -- one test threshold was initially wrong (tail_ratio on a two-point-mass distribution), traced to exact numpy percentile-interpolation arithmetic (1.0 + 0.01*999 = 10.99, matched the measured value precisely), and fixed in the TEST, not the function, then re-verified on an unambiguous lognormal case (114.51, correctly >>1). Plan: header amended with the headline (read-first), §7 RUN section appended in full. Board: STATUS_BOARD D-CZ-2..7 rows updated from Queued to their real outcomes; EPIPHANIES E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND- REVERSED-1 (prepend, suffix-verified). Honest scope stated in both homes: one box size, one variable (MSLP), three timesteps plus 19 storms. The geometry axis (the other half of "good geometry vs badly calibrated") was never tested -- C5's structural blocker means it couldn't be, at this box scale. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/EPIPHANIES.md | 78 +++ .claude/board/STATUS_BOARD.md | 16 +- .claude/plans/substrate-comfort-zones-v1.md | 177 ++++- .../exec-runs/substrate_comfort_d_cz_2_7.txt | 1 + .../substrate_comfort_d_cz_2_7.json | 628 ++++++++++++++++++ .../weather-p1/substrate_comfort_d_cz_2_7.py | 499 ++++++++++++++ 6 files changed, 1387 insertions(+), 12 deletions(-) create mode 100644 probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt create mode 100644 probes/weather-p1/substrate_comfort_d_cz_2_7.json create mode 100644 probes/weather-p1/substrate_comfort_d_cz_2_7.py diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index f63cb06bc..d97b448b4 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,81 @@ +## 2026-08-12 — E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND-REVERSED-1 + +**Status:** FINDING `[G]` — measured, both pre-registered bars, one run. +`probes/weather-p1/substrate_comfort_d_cz_2_7.json`, plan §7. + +**The hypothesis this whole plan existed to test is refuted, and refuted +in the SHARPEST available form: not "no effect", but a clean monotonic +reversal.** The operator's hypothesis — *a badly-calibrated substrate +that maps dynamically preserves MORE structure in strong storms than a +well-calibrated absolute one* — was tested two ways, pre-registered, on +an equal-budget cross-swap over four regimes (calm → ocean → active → +storm): + +- **C3 (transfer loss)** required `L̄[R4] < L̄[R1]`. Measured: + `L̄` = **0.011 → 0.309 → 0.671 → 0.690**, R1→R4 — a **62× increase**, + monotonic across all four tiers. Storms are *less* forgiving of a + foreign absolute calibration, not more. +- **C4 (the crossover)** required a sign flip against the diagonal — + absolute winning in calm, dynamic winning in storm. Measured: absolute + wins its own diagonal in **all four** regimes, and its margin **grows** + from ~1 Pa (R1/R2) to **10.78 Pa** (R4) — the opposite of a crossover. + +Both measures point the same direction. This is not two noisy nulls; it +is one coherent, reversed relationship measured twice by independent +instruments (`ρ` on rank vs RMSE on the diagonal). + +**A real construction bug was found and fixed en route, via the gate +designed to catch exactly this class of defect.** C0 — *"both controls +must be WORSE than every real arm in every regime; if either matches a +real arm anywhere, that cell measures nothing"* — failed on its first run +in R1 and R4: `GEO-DEGENERATE` was built from `truth[:len//64]`, the +first slice of an array already shuffled by `rng.choice` for equal +budget. A random subsample of a flat array is not reliably narrower in +range than the whole array; that is a materially different construction +from D-CZ-1's correct one (`p[si,sj][:n_i,:n_i]`, a genuine 2-D corner). +Fixed by carrying each regime's full 2-D box alongside the flat +evaluation sample and building the degenerate donor from a real spatial +corner. **Disable-verified**: reverting to the flat-slice construction +reproduces the exact original failure; the fix reproduces the identical +real-arm numbers (C2–C6 unchanged) while making C0 pass cleanly in all +four regimes. The bug lived entirely in the control; nothing about the +real arms was ever wrong. Recorded because it is this arc's C0 gate doing +precisely the job it was built for — catching a defect in the apparatus +before it could contaminate a verdict — on the very first probe that +tried to use it for something other than a smoke test. + +**And it contradicts an earlier exploratory hint, on purpose, stated +plainly rather than averaged away.** §6.6 measured, *within R4 only*, a +single arm's saturation correlating with storm `\|∇p\|` at ρ = +0.444 +(p = 0.058, explicitly labelled not significant, not a result). That +correlation's *direction* matched the hypothesis. The properly-powered, +pre-registered, cross-regime test says the opposite. Where an +unpre-registered n=19 single-regime correlation and a pre-registered +cross-swap disagree, **the pre-registered result governs** — and this +entry exists so a later read of the plan does not quietly split the +difference between "p=0.058, direction matches" and "reversed, +monotonic, two independent measures" into a false middle. + +**C1c passed and licensed the whole exercise** — the regimes measurably +differ in correlation structure (decay length, Gini, tail ratio of +`|∇p|`), not merely in gradient magnitude, so the refutation above is a +finding about calibration under real structural variation, not an +artifact of four copies of one condition. + +**What is NOT concluded.** One box size (16°, 4225 cells), one variable +(MSLP), three timesteps plus 19 storms. C5 — the *other* half of "good +geometry vs badly calibrated" — was never run: the golden index floor +(N ≥ 2,550,409) is three orders of magnitude above what a box this size +can hold, so `GEO-GOLDEN-HI` has no admissible construction at this +scale. The geometry axis of the operator's original framing remains +completely untested. + +**Cross-ref:** `E-A-HORSE-RACE-IS-NOT-A-CROSS-SWAP-1` (the instrument +this run finally exercised); `E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1` +(the exact defect class C0 caught here); `E-A-FIGURE-CITED-TWICE-IS-NOT- +CONFIRMED-ONCE-1` (why §6.6's direction is not treated as confirmed by +this section merely because it was written down once already). + ## 2026-08-12 — E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1 **Status:** FINDING `[G]` — measured. Probe diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 86b61340f..7afe6d922 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -35,14 +35,14 @@ already run and it corrected two regime definitions before any bar existed. |---|---|---|---| | D-CZ-0 | §1 regime preflight (`\|∇p\|` ladder, elevation-confound screen, speed-is-not-the-discriminator finding) | **DONE — and now REPRODUCED + partly corrected (§6.1/6.2).** It had NO committed script or JSON when marked DONE. 4 rows reproduce (1.004/1.022/0.994/0.931); the 5 EXCLUDED land candidates are **unreproducible** (centres never recorded). Definition identified: Pa per grid cell **without cos(lat)** — R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, order survives, range 9.3× → **9.8×**. Original text:** — ladder R1 Amazon 10.2 → R2 ocean 14.9 → R3 W Siberia 43.8 → R4 storm 95.6 (9.3× range); 4 land candidates excluded on elev σ > 150 m | the regime axis all other rows score on | | D-CZ-1 | C0 controls (shuffled codebook + degenerate geometry), losability-smoke-tested BEFORE the full run | **DONE — PASS** (`substrate_comfort_d_cz_0_1.py/.json`). Both controls lose to both real arms on BOTH metrics in ALL FOUR regimes; mechanism visible (`GEO-DEGENERATE` saturates **72–97 %** per regime — corrected 2026-08-12 from a stale 92–97 %, see plan §6.3; R4 is the low end at 0.7224). C1b `separation` = **6.28** vs the ≥ 3 bar. **AND the run amended C4**: `ρ` is SATURATED on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps `ρ` off-diagonal, C4 moves to RMSE in Pa (§6.4) | gates every cell — a control that can't lose voids its cell | -| D-CZ-2 | C1 regime-ladder stability across ≥3 timesteps | Queued | anti-cherry-pick on the whole regime axis | -| D-CZ-2b | **C1b constancy is measured** — `separation = (between-box range)/(mean within-box σ)` ≥ 3 | Queued | earns the phrase "held constant"; otherwise a caveat rides every cell | -| D-CZ-2c | **C1c the suitability ASSUMPTION** — regimes must differ in autocorrelation decay + rank-distribution shape, not only in `\|∇p\|` | Queued | a null here VOIDS the cross-swap reading (spread manufactured on the wrong axis) | -| D-CZ-3 | **C2 degenerate-row verification, both halves** — dynamic arms `L ≡ 0` exactly, AND `CAL-ABS` proven non-degenerate through the same code path | Queued | the can-it-DIFFER gate; without half (ii) a flat row proves nothing | -| D-CZ-4 | C3 **transfer loss** vs turbulence — `L̄[T] = mean_{D≠T} L[D][T]` strictly smaller in R4 than R1, reported with `occupancy`/`saturation` | Queued | "storms forgive bad calibration", with a mechanism attached | -| D-CZ-5 | **C4 the crossover** — `ρ(CAL-RANK) − ρ(CAL-ABS, D=T)` sign flip calm↔storm (vs the DIAGONAL, the hardest opponent) | Queued | **the operator's hypothesis, two-sided**; weak form vs `D≠T` reported separately | -| D-CZ-6 | C5 geometry floor on a SAMPLING-fidelity metric (NULL expected-plausible per W5 B4) | Queued | does the index floor bite where smoothing didn't | -| D-CZ-7 | C6 the transfer matrix (the deliverable) | Queued | comfort read off the diagonal, travel-cost off the off-diagonal | +| D-CZ-2 | C1 regime-ladder stability across ≥3 timesteps | **DONE — PASS.** `\|∇p\|` order R1 **2026-08-12 — rows re-cut, not restated.** The operator ruled that the > plan's §2 was built as a horse race where a cross-swap diagnostic belongs diff --git a/.claude/plans/substrate-comfort-zones-v1.md b/.claude/plans/substrate-comfort-zones-v1.md index 592402f03..241818cb8 100644 --- a/.claude/plans/substrate-comfort-zones-v1.md +++ b/.claude/plans/substrate-comfort-zones-v1.md @@ -1,11 +1,21 @@ # substrate-comfort-zones-v1 — where does each substrate formula feel at home? -> **Status:** ACTIVE, exploratory tier. Bars author-written and unaudited — -> fine for exploratory, said out loud (the -> `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` rule). -> No verdict-tier claim may be promoted out of this plan without an +> **Status:** RUN, §7 — the pre-registered hypothesis is **REFUTED** on this +> data. Bars author-written and unaudited — fine for exploratory, said out +> loud (the `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` +> rule). No verdict-tier claim may be promoted out of this plan without an > independent adversarial spec audit. > +> **⚠ HEADLINE, READ FIRST — see §7 for the full run.** Both pre-registered +> measures of the hypothesis below point the SAME direction, and it is the +> OPPOSITE of what was hypothesized: `C3`'s transfer loss `L̄` grows +> monotonically 0.011 → 0.309 → 0.671 → **0.690** from calm to storm (not +> shrinks); `C4`'s crossover never flips — absolute wins its own diagonal +> in every regime, by a MARGIN THAT GROWS from ~1 Pa to **10.78 Pa** in the +> storm regime, not shrinks or reverses. On this box, this variable +> (MSLP), these timesteps: **a well-calibrated absolute encoding does not +> lose its edge under turbulence — it gains one.** +> > **Operator framing (2026-08-12), three messages:** > 1. *hold different situations constant — over water vs flatland vs storm > with high velocity differences / turbulence* @@ -624,3 +634,162 @@ control was run on this particular comparison; and the whole point of the cross-swap design is that a diagonal-vs-diagonal difference is not what the hypothesis is about. Recorded so a later run cannot present it as a confirmation that was there all along. + + +--- + +## §7 RUN — D-CZ-2..7, the cross-swap matrix (2026-08-12) + +Script `probes/weather-p1/substrate_comfort_d_cz_2_7.py`, results +`…_d_cz_2_7.json`, tag-file `exec-runs/…txt`. Run in the order the plan +requires: **C1c first**, because a null there would VOID the interpretation +before anything downstream is worth trusting. C0's construction had a real +bug, found and disable-verified fixed before any of C2–C6 was reported. + +### §7.0 A construction bug in C0 itself, caught by C0's own gate + +The first run of this probe **failed** C0 in two of four regimes: +`GEO-DEGENERATE` did not lose to the real arms in R1 or R4. Per §3's own +rule — *"if either matches a real arm anywhere, that cell measures +nothing and is reported VOID"* — nothing downstream could have been +trusted as written. + +**Cause:** the "degenerate donor" was built from `truth[:len//64]` — the +first slice of an array already subsampled by `rng.choice` for equal +budget. `rng.choice` returns no spatial order, so that slice is an +**ordinary random subsample**, not a narrow spatial patch — a materially +different construction from D-CZ-1's correct one +(`p[si,sj][:n_i,:n_i]`, a genuine 2-D corner). A random subsample of a +flat array is not reliably narrower in range than the whole array; by +chance it can nearly match it. + +**Fixed** by keeping each regime's full 2-D box alongside the equal-budget +flat evaluation sample, and building the degenerate donor from an actual +`n_i × n_i` corner of that box (`n_i = side // 8`), matching D-CZ-1 +exactly. **Disable-verified**: reverting to the flat-slice construction +reproduces the *exact* original failure (R1 and R4 fail, R2/R3 pass); +the fix reproduces the same real-arm numbers unchanged (C2–C6 identical +between the two runs) while making **C0 pass cleanly in all four +regimes**. The bug was isolated to the control; the real arms were never +wrong. + +### §7.1 C1c PASSES — the cross-swap interpretation is licensed + +| regime | decay length (cells) | Gini(\|∇p\|) | tail ratio (p99/p50) | +|---|---|---|---| +| R1 CALM | 25.5 | 0.4321 | **7.65** | +| R2 OCEAN | 32.0 | 0.2758 | 3.72 | +| R3 ACTIVE | 32.0 | 0.2262 | 1.94 | +| R4 STORM (median of 19) | 22.5 | 0.3062 | **2.95** | + +R4/R1 ratios: decay **0.88**, Gini **0.71**, tail ratio **0.385** — the +storm regime's gradient field is far *less* concentrated (a smaller +handful of cells carrying most of the gradient in calm regions; a more +uniformly elevated field in storms) than the calm regime's, a genuinely +different **shape**, not just a different mean. All three deviate from 1 +by ≥ 20 %. **The regimes differ in structure, not merely in `\|∇p\|`** — +the assumption behind the whole regime ladder holds. + +*(The 20 % threshold is this run's own operationalization of §3's +qualitative bar — the plan did not pre-register a number. Flagged as an +author choice; the raw ratios are reported above so a reader can apply a +stricter one and reach the same conclusion — the smallest deviation, +12 %, is on decay length, but tail ratio's 61 % deviation alone clears +any reasonable bar.)* + +**C1 (ladder) and C1b (separation) both PASS too**: the `\|∇p\|` order +R1 < R2 < R3 < R4 holds at all three tested timesteps, and separation is +**5.87–8.24** at every one — comfortably above the ≥ 3 bar. + +### §7.2 C2 PASSES both halves + +Dynamic arms (`CAL-RANK`, `CAL-FISHERZ`) show `L[D][T] = 0` **exactly** +for every donor in every regime — no leak. Through the *identical* code +path, `CAL-ABS`'s off-diagonal `L` ranges **0.011 (R1) → 0.947 (R4)** — +demonstrably non-degenerate. The can-it-DIFFER gate is satisfied. + +### §7.3 ⚠ C3 FAILS — and the failure is a clean, monotonic REVERSAL + +`L̄[T]`, the mean off-diagonal transfer loss for `CAL-ABS`: + +| R1 CALM | R2 OCEAN | R3 ACTIVE | R4 STORM | +|---|---|---|---| +| **0.011** | 0.309 | 0.671 | **0.690** | + +The bar required `L̄[R4] < L̄[R1]`. **Measured: `L̄[R4]` is 62× larger than +`L̄[R1]`, and the increase is monotonic across all four tiers.** This is +not "no shrinkage" — it is the pre-registered relationship holding in +**reverse**: transfer loss under a foreign absolute calibration grows, +not shrinks, as turbulence rises. Storms are **less** forgiving of bad +calibration on this data, not more. + +### §7.4 ⚠ C4 FAILS — same direction, same shape + +`Δ[T] = RMSE(CAL-RANK, own window) − RMSE(CAL-ABS, own diagonal)`, in Pa +(positive = absolute wins): + +| R1 CALM | R2 OCEAN | R3 ACTIVE | R4 STORM | +|---|---|---|---| +| +1.29 | +1.11 | +0.46 | **+10.78** | + +**No sign flip anywhere** — absolute wins its own diagonal in all four +regimes, and its margin of victory is **largest in the storm regime**, +not smallest. The strong form of the operator's hypothesis is refuted on +this data: a well-calibrated absolute encoding does not lose its edge as +turbulence rises here — it gains one. + +The **weak form** (`ρ(CAL-RANK) > ρ(CAL-ABS, D ≠ T)`) passes trivially +everywhere, exactly as pre-registered it would — dynamic beats a +genuinely *mis-calibrated* absolute by **329–2508 Pa** across regimes. +Per §3: *"nearly guaranteed by C2(i)… evidence of wiring, not of merit."* +Reported, not conflated with C4's real bar. + +### §7.5 C5 — NOT RUN, structural blocker + +`GEO-GOLDEN-HI` needs `N ≥ F(17)² = 2 550 409` before the golden lattice +behaves like the irrational it approximates (§1's own index-floor rule). +A 16° box at 0.25° resolution holds `65 × 65 = 4225` cells — **three +orders of magnitude below the floor**. `GEO-GOLDEN-HI` cannot be +constructed at box scale on this grid; the comparison has no admissible +high arm. Reported as a structural gap, not skipped silently, and not +faked with an interpolated sub-grid lattice (which would sample the +interpolator, not the field). + +### §7.6 C6 — the transfer matrix, delivered + +Full `donor × target` table for all five arms (`CAL-ABS`, `CAL-RANK`, +`CAL-FISHERZ`, `CAL-SHUFFLE`, `GEO-DEGENERATE`) × four regimes, every +`ρ` / `L` / `occupancy` / `saturation` / RMSE / bias cell raw, in +`substrate_comfort_d_cz_2_7.json` → `C6_matrix`. + +### §7.7 ⚠ This CONTRADICTS §6.6's exploratory hint — stated plainly + +§6.6 measured, *within* R4 only, ρ(`GEO-DEGENERATE` ρ, storm `\|∇p\|`) = ++0.444 (p = 0.058, not significant) — the *direction* of that correlation +matched the operator's hypothesis, and was recorded as a coherence with +the D-CZ-1 saturation numbers. + +**The properly-powered, cross-regime test says the opposite.** §6.6 was a +single-regime, single-arm, unpre-registered correlation at n=19. §7.3/§7.4 +are the pre-registered, cross-swap, equal-budget bars the whole plan was +built to produce. Where they disagree, the pre-registered cross-regime +result governs. §6.6 is retained as what it always was — exploratory, not +a result — and this section exists so a later read does not average the +two into a false middle. **The hypothesis is refuted on this data.** + +### §7.8 What this plan concludes + +The operator's hypothesis — *a badly-calibrated substrate that maps +DYNAMICALLY preserves MORE structure in strong storms than a +well-calibrated absolute one* — is **refuted, cleanly, on this box, this +variable, these three timesteps plus 19 storms**. Both pre-registered +measures of it (C3's transfer loss, C4's diagonal crossover) point the +same direction, monotonically, not merely "no effect." + +**What is NOT concluded:** that miscalibration is never more forgivable +under turbulence, anywhere, on any variable, at any scale — this is one +box size, one variable (MSLP), one gradient definition, and C5's absence +means the geometry axis (the *other* half of "good geometry vs badly +calibrated") was never tested at all. The regime axis and the C0/C1/C1b/ +C1c apparatus all hold; the specific calibration hypothesis they were +built to test does not. diff --git a/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt b/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt new file mode 100644 index 000000000..6198bc369 --- /dev/null +++ b/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt @@ -0,0 +1 @@ +DONE d-cz-2..7 diff --git a/probes/weather-p1/substrate_comfort_d_cz_2_7.json b/probes/weather-p1/substrate_comfort_d_cz_2_7.json new file mode 100644 index 000000000..bd3365a76 --- /dev/null +++ b/probes/weather-p1/substrate_comfort_d_cz_2_7.json @@ -0,0 +1,628 @@ +{ + "probe": "substrate_comfort_d_cz_2_7", + "seed": 20260812, + "timesteps": [ + 54358, + 55086, + 55818 + ], + "budget_cells_per_regime": 4225, + "units": { + "grad_p": "Pa/cell (flat, no cos-lat \u2014 the D-CZ-0 identified definition)", + "rmse": "Pa", + "bias": "Pa", + "decay_length": "grid cells", + "L": "dimensionless (rho difference)" + }, + "C0_controls": { + "R1_CALM": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R2_OCEAN": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R3_ACTIVE": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R4_STORM": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + } + }, + "C1_ladder": { + "per_timestep": { + "54358": { + "values": [ + 10.267982482910156, + 15.295616149902344, + 43.52193832397461, + 88.98800578870271 + ], + "ordered": true + }, + "55086": { + "values": [ + 8.420635223388672, + 20.382078170776367, + 21.0435848236084, + 88.98800578870271 + ], + "ordered": true + }, + "55818": { + "values": [ + 9.359210014343262, + 13.816144943237305, + 58.70938491821289, + 88.98800578870271 + ], + "ordered": true + } + }, + "holds_all": true, + "raw": { + "R1_CALM": { + "54358": 10.267982482910156, + "55086": 8.420635223388672, + "55818": 9.359210014343262 + }, + "R2_OCEAN": { + "54358": 15.295616149902344, + "55086": 20.382078170776367, + "55818": 13.816144943237305 + }, + "R3_ACTIVE": { + "54358": 43.52193832397461, + "55086": 21.0435848236084, + "55818": 58.70938491821289 + }, + "R4_STORM": { + "per_storm_mean": 88.98800578870271, + "per_storm_min": 39.901527404785156, + "n_independent_timesteps": 19 + } + } + }, + "C1b_separation": { + "per_timestep": { + "54358": 6.430169158055194, + "55086": 8.240830494144255, + "55818": 5.866366164820492 + }, + "bar": 3.0, + "holds_all": true + }, + "C1c_structure": { + "per_regime": { + "R1_CALM": { + "decay_length_cells": 25.5, + "gini_gradp": 0.4320934781503374, + "tail_ratio_gradp": 7.652638836611052 + }, + "R2_OCEAN": { + "decay_length_cells": 32.0, + "gini_gradp": 0.27581184507019185, + "tail_ratio_gradp": 3.722250017585821 + }, + "R3_ACTIVE": { + "decay_length_cells": 32.0, + "gini_gradp": 0.2262258291411534, + "tail_ratio_gradp": 1.9386696980814633 + }, + "R4_STORM": { + "decay_length_cells": 22.5, + "gini_gradp": 0.3062475714591397, + "tail_ratio_gradp": 2.9491883985991496, + "n": 19 + } + }, + "verdict": { + "decay_ratio_r4_over_r1": 0.8823529411764706, + "gini_ratio_r4_over_r1": 0.7087530521638368, + "tail_ratio_r4_over_r1": 0.3853818874203122, + "distinguishable": true + } + }, + "C2_degenerate_row": { + "dynamic_rows_flat": { + "CAL-RANK": { + "R1_CALM": 0.0, + "R2_OCEAN": 0.0, + "R3_ACTIVE": 0.0, + "R4_STORM": 0.0 + }, + "CAL-FISHERZ": { + "R1_CALM": 0.0, + "R2_OCEAN": 0.0, + "R3_ACTIVE": 0.0, + "R4_STORM": 0.0 + } + }, + "cal_abs_can_differ": { + "R1_CALM": 0.01103689701168109, + "R2_OCEAN": 0.6183197101240259, + "R3_ACTIVE": 0.8343471141874098, + "R4_STORM": 0.9467064832844216 + }, + "cal_abs_undefined_donors": { + "R1_CALM": [ + "R2_OCEAN", + "R3_ACTIVE" + ], + "R2_OCEAN": [ + "R1_CALM" + ], + "R3_ACTIVE": [ + "R1_CALM" + ], + "R4_STORM": [] + }, + "half_i_dynamic_L_is_exactly_zero": true, + "half_ii_cal_abs_nonzero_somewhere": true + }, + "C3_transfer_loss": { + "per_regime": { + "R1_CALM": { + "L_bar": 0.01103689701168109, + "n_defined_offdiag": 1, + "n_undefined_offdiag": 2, + "occupancy_mean_offdiag": 0.022135416666666668, + "saturation_mean_offdiag": 0.6666666666666666 + }, + "R2_OCEAN": { + "L_bar": 0.3094828671426827, + "n_defined_offdiag": 2, + "n_undefined_offdiag": 1, + "occupancy_mean_offdiag": 0.08463541666666667, + "saturation_mean_offdiag": 0.6609072978303747 + }, + "R3_ACTIVE": { + "L_bar": 0.6708209545588866, + "n_defined_offdiag": 2, + "n_undefined_offdiag": 1, + "occupancy_mean_offdiag": 0.10026041666666667, + "saturation_mean_offdiag": 0.9674161735700197 + }, + "R4_STORM": { + "L_bar": 0.6899068233087143, + "n_defined_offdiag": 3, + "n_undefined_offdiag": 0, + "occupancy_mean_offdiag": 0.3411458333333333, + "saturation_mean_offdiag": 0.9686785009861932 + } + }, + "r1_r4_both_defined": true, + "holds_r4_lt_r1": false + }, + "C4_crossover": { + "per_regime": { + "R1_CALM": { + "delta_rmse_rank_minus_absdiag": 1.2875496387398329, + "rho_floor_check_min": 0.99991523929089, + "weak_form_absforeign_minus_rank_rmse": 439.45736416501524 + }, + "R2_OCEAN": { + "delta_rmse_rank_minus_absdiag": 1.110984110781117, + "rho_floor_check_min": 0.999916685049489, + "weak_form_absforeign_minus_rank_rmse": 328.7326356721833 + }, + "R3_ACTIVE": { + "delta_rmse_rank_minus_absdiag": 0.4552001128419074, + "rho_floor_check_min": 0.9999168206445191, + "weak_form_absforeign_minus_rank_rmse": 1635.2923826909162 + }, + "R4_STORM": { + "delta_rmse_rank_minus_absdiag": 10.780337293854465, + "rho_floor_check_min": 0.9999168302772791, + "weak_form_absforeign_minus_rank_rmse": 2507.545438988307 + } + }, + "sign_flip": false + }, + "C6_matrix": { + "R1_CALM": { + "CAL-ABS|R1_CALM": { + "rho": 0.9999596603993293, + "rmse_pa": 0.43545659099220974, + "bias_pa": -0.0039270091762204145, + "occupancy": 0.8359375, + "saturation": 0.00047337278106508875 + }, + "CAL-ABS|R2_OCEAN": { + "rho": NaN, + "rmse_pa": 245.86851734876652, + "bias_pa": 237.65027211567354, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": NaN, + "rmse_pa": 1069.5368025036043, + "bias_pa": 1067.677524312939, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9889227633876482, + "rmse_pa": 8.135791331870866, + "bias_pa": 0.19725649500739645, + "occupancy": 0.05859375, + "saturation": 0.0 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.08232528142572652, + "rmse_pa": 140.96730416561363, + "bias_pa": 59.71426860380455, + "occupancy": 0.8359375, + "saturation": 0.00047337278106508875 + }, + "GEO-DEGENERATE|self": { + "rho": 0.3799822353435944, + "rmse_pa": 119.99599105198327, + "bias_pa": -103.18710295727972, + "occupancy": 0.421875, + "saturation": 0.949585798816568 + } + }, + "R2_OCEAN": { + "CAL-ABS|R1_CALM": { + "rho": NaN, + "rmse_pa": 525.9022909934547, + "bias_pa": -463.10123923758783, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.9999805198027167, + "rmse_pa": 1.3133243896042126, + "bias_pa": 0.02958709175064719, + "occupancy": 0.9609375, + "saturation": 0.0009467455621301775 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.3816608096786909, + "rmse_pa": 445.52070839237126, + "bias_pa": 376.5969895438471, + "occupancy": 0.1171875, + "saturation": 0.9488757396449704 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9993344956413772, + "rmse_pa": 22.047833131879898, + "bias_pa": -2.914080875554734, + "occupancy": 0.1328125, + "saturation": 0.033846153846153845 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": -0.04867588197240352, + "rmse_pa": 452.1981665483234, + "bias_pa": 122.80125530897513, + "occupancy": 0.9609375, + "saturation": 0.0009467455621301775 + }, + "GEO-DEGENERATE|self": { + "rho": 0.47801853597161725, + "rmse_pa": 430.5506483061611, + "bias_pa": -354.5808563629692, + "occupancy": 0.6484375, + "saturation": 0.917396449704142 + } + }, + "R3_ACTIVE": { + "CAL-ABS|R1_CALM": { + "rho": NaN, + "rmse_pa": 2258.161742708387, + "bias_pa": -2129.5621933795996, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.4926944507153999, + "rmse_pa": 1224.020331977599, + "bias_pa": -979.967107865373, + "occupancy": 0.28125, + "saturation": 0.9114792899408284 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.9999892456457632, + "rmse_pa": 3.2888446705759047, + "bias_pa": -0.1164641358866494, + "occupancy": 1.0, + "saturation": 0.002366863905325444 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.1656421314583534, + "rmse_pa": 1434.9272077370156, + "bias_pa": -1223.0108273622411, + "occupancy": 0.015625, + "saturation": 0.9907692307692307 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.1076028284432391, + "rmse_pa": 1056.1259072749588, + "bias_pa": 151.59143322103827, + "occupancy": 1.0, + "saturation": 0.002366863905325444 + }, + "GEO-DEGENERATE|self": { + "rho": 0.2900859787206164, + "rmse_pa": 1594.7041920434797, + "bias_pa": 1408.846735254773, + "occupancy": 0.3359375, + "saturation": 0.9718343195266272 + } + }, + "R4_STORM": { + "CAL-ABS|R1_CALM": { + "rho": 0.5072287551023801, + "rmse_pa": 2070.4683206201353, + "bias_pa": 1647.7540435321514, + "occupancy": 0.56640625, + "saturation": 0.954792899408284 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.36970751728458107, + "rmse_pa": 2388.553976512638, + "bias_pa": 2021.596971074392, + "occupancy": 0.4375, + "saturation": 0.952189349112426 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.053268646229919614, + "rmse_pa": 3119.8878118239486, + "bias_pa": 2838.3526022068, + "occupancy": 0.01953125, + "saturation": 0.9990532544378699 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9999751295143413, + "rmse_pa": 7.977593370078924, + "bias_pa": 0.10601527829142011, + "occupancy": 0.94140625, + "saturation": 0.0009467455621301775 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.006699959537369512, + "rmse_pa": 2536.695394356023, + "bias_pa": -603.1110531850961, + "occupancy": 0.94140625, + "saturation": 0.0009467455621301775 + }, + "GEO-DEGENERATE|self": { + "rho": 0.9038053970526956, + "rmse_pa": 1196.4020646066267, + "bias_pa": 420.62941414917714, + "occupancy": 0.921875, + "saturation": 0.8435502958579881 + } + } + }, + "c5_structural_blocker": "C5 (GEO-GOLDEN-LO vs GEO-GOLDEN-HI) is NOT RUN, and the reason is structural rather than budgetary. The golden index floor requires N >= F(17)^2 = 2,550,409 lattice points before the emergent parastichy stride behaves like phi at all. A 16-degree box at 0.25 degree resolution contains 65 x 65 = 4,225 cells -- three orders of magnitude below the floor. GEO-GOLDEN-HI therefore CANNOT be constructed at box scale on this grid, so the comparison the bar asks for has no admissible high arm. Reported rather than faked with an interpolated sub-grid lattice, which would sample the interpolator and not the field." +} \ No newline at end of file diff --git a/probes/weather-p1/substrate_comfort_d_cz_2_7.py b/probes/weather-p1/substrate_comfort_d_cz_2_7.py new file mode 100644 index 000000000..326791d68 --- /dev/null +++ b/probes/weather-p1/substrate_comfort_d_cz_2_7.py @@ -0,0 +1,499 @@ +"""D-CZ-2..7 — the cross-swap transfer matrix and the gates that precede it. + +Runs the bars pre-registered in `.claude/plans/substrate-comfort-zones-v1.md` +§3, in the order the plan itself requires: **C1c first**, because a null there +VOIDS the cross-swap interpretation rather than weakening it. + + C1 regime ladder holds on >= 3 independent timesteps + C1b separation = (between-box range) / (mean within-box sigma) >= 3 + C1c the regimes must differ in CORRELATION STRUCTURE, not merely |grad p| + C2 the degenerate row -- both halves (dynamic L == 0 exactly, AND + CAL-ABS proven able to be non-degenerate through the same code path) + C3 transfer loss shrinks with turbulence + C4 the crossover, on the AMENDED metric (RMSE in Pa, not rho -- §6.4) + C6 the full transfer matrix, every cell raw + +C5 (the geometry axis) is NOT run here, and the reason is structural rather +than budgetary -- see `c5_structural_blocker` in the JSON. It is reported, +not skipped silently. + +EQUAL BUDGET is enforced by construction: every regime is subsampled to the +same cell count with a seeded RNG before any codebook is built. The W2s-a +lesson (an unequal budget silently advantages the arm with more samples) is +not re-learned here. + +R4's timesteps are the 19 storms' OWN t0 values -- measuring a storm box at +an unrelated hour is the defect this probe's predecessor already caught and +fixed. That also means R4 satisfies C1's ">= 3 independent timesteps" with +19, while R1..R3 are measured at 3 explicit timesteps each. +""" + +import datetime +import json +import pathlib +import urllib.request + +import numcodecs +import numpy as np + +SEED = 20260812 +B = ("https://storage.googleapis.com/weatherbench2/datasets/era5/" + "1959-2022-6h-1440x721.zarr") +R_E = 6371.0 +BOX_DEG = 16.0 +N_LEVELS = 256 +G0 = 9.80665 + +# C1 needs >= 3 independent timesteps. t0 is the preflight's; the other two are +# offset by ~6 months and ~1 year so the ladder is tested across seasons, not +# across three adjacent hours (which would be one condition sampled thrice). +TIMESTEPS = [54358, 54358 + 4 * 182, 54358 + 4 * 365] + +HERE = pathlib.Path(__file__).parent +OUT = HERE / "substrate_comfort_d_cz_2_7.json" +TAG = HERE / "exec-runs" / "substrate_comfort_d_cz_2_7.txt" + +op = urllib.request.build_opener(urllib.request.ProxyHandler({})) +meta = json.loads(op.open(B + "/.zmetadata", timeout=90).read())["metadata"] +EPOCH = datetime.datetime(1959, 1, 1) + + +def t_index(dt): + """WB2 time index: 6-hourly steps since 1959-01-01. Anchor guard.""" + return int(round((dt - EPOCH).total_seconds() / 3600 / 6)) + + +assert t_index(datetime.datetime(2021, 6, 15, 12)) == 91246 + + +def fetch(var, key): + """Fetch and decode one zarr chunk from the WB2 store.""" + za = meta[f"{var}/.zarray"] + raw = op.open(f"{B}/{var}/{key}", timeout=900).read() + dec = numcodecs.get_codec(za["compressor"]).decode(raw) + return np.frombuffer(dec, dtype=np.dtype(za["dtype"])).reshape(za["chunks"]) + + +LATS = np.linspace(90.0, -90.0, 721) +LONS = np.linspace(0.0, 359.75, 1440) + + +def box_index(clat, clon, deg=BOX_DEG): + """Row indices and WRAPPED column indices for a deg x deg box.""" + i0 = int(np.argmin(np.abs(LATS - (clat + deg / 2)))) + i1 = int(np.argmin(np.abs(LATS - (clat - deg / 2)))) + jc = int(np.argmin(np.abs(LONS - (clon % 360)))) + half = int(round((deg / 2) / 0.25)) + return np.arange(i0, i1 + 1), (np.arange(jc - half, jc + half + 1)) % len(LONS) + + +def box(field, rows, cols): + """Extract a box from a 2-D field, honouring the longitude wrap.""" + return field[np.ix_(rows, cols)] + + +def grad_flat(p, rows, cols): + """|grad p| in Pa per grid cell, NO cos(lat) — the definition identified + from the data in D-CZ-0 as the one the §1 preflight figures used. Kept so + the ladder here is comparable to the recorded table, not to a different + quantity.""" + sub = box(p, rows, cols) + return np.hypot(np.gradient(sub, axis=1), np.gradient(sub, axis=0)) + + +def spearman(a, b): + """Spearman rho via Pearson on ranks, ties averaged; nan on constant + input (rho is undefined there, not zero).""" + def rank(x): + order = np.argsort(x, kind="stable") + r = np.empty(len(x), float) + r[order] = np.arange(len(x), dtype=float) + xs = x[order] + i = 0 + while i < len(xs): + j = i + while j + 1 < len(xs) and xs[j + 1] == xs[i]: + j += 1 + if j > i: + r[order[i:j + 1]] = (i + j) / 2.0 + i = j + 1 + return r + ra, rb = rank(np.asarray(a, float)), rank(np.asarray(b, float)) + if ra.std() == 0 or rb.std() == 0: + return float("nan") + return float(np.corrcoef(ra, rb)[0, 1]) + + +# ------------------------------------------------------- C1c instruments + +def decay_length_cells(field2d): + """Autocorrelation decay length in grid cells: the lag at which the + field's spatial autocorrelation first drops below 1/e, averaged over the + two axes. Measured on the ANOMALY (box mean removed) so the decay is of + structure, not of the offset.""" + a = field2d - field2d.mean() + out = [] + for axis in (0, 1): + n = a.shape[axis] + cors = [] + for lag in range(1, min(n // 2, 40)): + x = np.take(a, range(0, n - lag), axis=axis) + y = np.take(a, range(lag, n), axis=axis) + xv, yv = x.ravel(), y.ravel() + if xv.std() == 0 or yv.std() == 0: + cors.append(0.0) + else: + cors.append(float(np.corrcoef(xv, yv)[0, 1])) + thr = 1.0 / np.e + hit = next((i + 1 for i, c in enumerate(cors) if c < thr), len(cors) + 1) + out.append(float(hit)) + return float(np.mean(out)) + + +def gini(x): + """Gini coefficient of a non-negative sample — the concentration of the + |grad p| distribution. 0 = every cell equally gradient-y, 1 = all the + gradient in one cell.""" + v = np.sort(np.asarray(x, float).ravel()) + v = v[np.isfinite(v)] + if v.size == 0 or v.sum() == 0: + return float("nan") + n = v.size + idx = np.arange(1, n + 1) + return float((2 * (idx * v).sum()) / (n * v.sum()) - (n + 1) / n) + + +def tail_ratio(x): + """p99 / p50 of |grad p| — how far the extreme sits above the typical.""" + v = np.asarray(x, float).ravel() + med = float(np.percentile(v, 50)) + return float(np.percentile(v, 99) / med) if med > 0 else float("nan") + + +# ------------------------------------------------------------- the codecs + +def encode_decode(values, lo, hi, decode_table=None): + """Quantize to N_LEVELS uniform levels over [lo, hi], decode to the level + centre. Values outside SATURATE — the mechanism a wrong donor fails by.""" + lo, hi = float(lo), float(hi) + if hi <= lo: + hi = lo + 1e-9 + width = (hi - lo) / N_LEVELS + idx = np.clip(((values - lo) / width).astype(int), 0, N_LEVELS - 1) + centres = lo + (np.arange(N_LEVELS) + 0.5) * width + if decode_table is not None: + centres = centres[decode_table] + return centres[idx], idx + + +def rank_codec(values): + """CAL-RANK: 256 quantile levels re-derived INSIDE the window. No donor + exists — that is the property under test, not a defect.""" + qs = np.quantile(values, np.linspace(0, 1, N_LEVELS + 1)) + idx = np.clip(np.searchsorted(qs, values, side="right") - 1, 0, N_LEVELS - 1) + return 0.5 * (qs[:-1] + qs[1:])[idx], idx + + +def fisherz_codec(values): + """CAL-FISHERZ: uniform levels in arctanh(2*rank-1) — resolution moved + into the tails. Also window-local, so also donor-free.""" + n = len(values) + r = (np.argsort(np.argsort(values)) + 0.5) / n + s = np.clip(2 * r - 1, -0.999999, 0.999999) + z = np.arctanh(s) + dec_z, idx = encode_decode(z, z.min(), z.max()) + # map the decoded z back to a value by inverting through the sorted sample + back = np.clip((np.tanh(dec_z) + 1) / 2, 0, 1) + srt = np.sort(values) + pos = np.clip((back * n).astype(int), 0, n - 1) + return srt[pos], idx + + +def cell_metrics(dec, idx, truth): + """One matrix cell's numbers. Every quantity the plan's C6 requires.""" + resid = dec - truth + return { + "rho": spearman(dec, truth), + "rmse_pa": float(np.sqrt(np.mean(resid ** 2))), + "bias_pa": float(np.mean(resid)), + "occupancy": float(len(np.unique(idx)) / N_LEVELS), + "saturation": float(np.mean((idx == 0) | (idx == N_LEVELS - 1))), + } + + +# ------------------------------------------------------------------- main + +def main(): + """Run C1c first (it can void everything), then C1/C1b, then the matrix.""" + TAG.parent.mkdir(exist_ok=True) + TAG.write_text("START d-cz-2..7\n") + rng = np.random.default_rng(SEED) + + named = { + "R1_CALM": (-4.0, 296.0), + "R2_OCEAN": (-25.0, 220.0), + "R3_ACTIVE": (60.0, 72.0), + } + f14 = json.loads((HERE / "comet_tail_f14.json").read_text()) + qual = [r for r in f14["rows"] + if r.get("status") == "OK" and r.get("displacement_km", 0) >= 250] + assert len(qual) == f14["n_qualifying"] == 19 + + # ---- fields ---- + fields = {t: fetch("mean_sea_level_pressure", f"{t}.0.0")[0] for t in TIMESTEPS} + storm_fields = {} + for r in qual: + t = int(r["t0"]) + if t not in storm_fields: + storm_fields[t] = fetch("mean_sea_level_pressure", f"{t}.0.0")[0] + + # ---- C1 / C1b : ladder + separation across timesteps ---- + ladder, within = {}, {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + ladder[name] = {str(t): float(grad_flat(fields[t], rr, cc).mean()) + for t in TIMESTEPS} + within[name] = {str(t): float(grad_flat(fields[t], rr, cc).std()) + for t in TIMESTEPS} + storm_g = [] + for r in qual: + rr, cc = box_index(float(r["center_lat"]), float(r["center_lon"])) + storm_g.append(float(grad_flat(storm_fields[int(r["t0"])], rr, cc).mean())) + ladder["R4_STORM"] = {"per_storm_mean": float(np.mean(storm_g)), + "per_storm_min": float(np.min(storm_g)), + "n_independent_timesteps": len(set(int(r["t0"]) for r in qual))} + + c1 = {} + for t in TIMESTEPS: + vals = [ladder["R1_CALM"][str(t)], ladder["R2_OCEAN"][str(t)], + ladder["R3_ACTIVE"][str(t)], float(np.mean(storm_g))] + c1[str(t)] = {"values": vals, "ordered": bool(all( + vals[i] < vals[i + 1] for i in range(3)))} + c1_holds = all(v["ordered"] for v in c1.values()) + + c1b = {} + for t in TIMESTEPS: + means = [ladder[k][str(t)] for k in named] + [float(np.mean(storm_g))] + sig = float(np.mean([within[k][str(t)] for k in named])) + c1b[str(t)] = float((max(means) - min(means)) / sig) + + # ---- C1c : do the regimes differ in STRUCTURE, not just in |grad p|? ---- + c1c = {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + f = box(fields[TIMESTEPS[0]], rr, cc) + g = grad_flat(fields[TIMESTEPS[0]], rr, cc) + c1c[name] = {"decay_length_cells": decay_length_cells(f), + "gini_gradp": gini(g), "tail_ratio_gradp": tail_ratio(g)} + dl, gi, tr = [], [], [] + for r in qual: + rr, cc = box_index(float(r["center_lat"]), float(r["center_lon"])) + f = box(storm_fields[int(r["t0"])], rr, cc) + g = grad_flat(storm_fields[int(r["t0"])], rr, cc) + dl.append(decay_length_cells(f)); gi.append(gini(g)); tr.append(tail_ratio(g)) + c1c["R4_STORM"] = {"decay_length_cells": float(np.median(dl)), + "gini_gradp": float(np.median(gi)), + "tail_ratio_gradp": float(np.median(tr)), + "n": len(dl)} + # the bar: R1 and R4 must be DISTINGUISHABLE on these + r1, r4 = c1c["R1_CALM"], c1c["R4_STORM"] + c1c_verdict = { + "decay_ratio_r4_over_r1": r4["decay_length_cells"] / r1["decay_length_cells"], + "gini_ratio_r4_over_r1": r4["gini_gradp"] / r1["gini_gradp"], + "tail_ratio_r4_over_r1": r4["tail_ratio_gradp"] / r1["tail_ratio_gradp"], + } + c1c_verdict["distinguishable"] = bool( + max(abs(v - 1.0) for k, v in c1c_verdict.items() if k.endswith("r1")) >= 0.20) + + # ---- equal-budget samples, one per regime -- AND a genuine spatial + # sub-patch per regime for GEO-DEGENERATE, kept SEPARATE from the + # equal-budget flat sample. Slicing the flat (post-subsample, post- + # shuffle) array is NOT a spatial sub-patch -- rng.choice returns no + # spatial order, so its first N elements are an ordinary random + # subsample, not a narrow corner. The degenerate donor must be built + # from the box's own 2-D layout, exactly as D-CZ-1 does it. + boxes2d = {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + boxes2d[name] = box(fields[TIMESTEPS[0]], rr, cc).astype(float) + storm_boxes2d = [ + box(storm_fields[int(r["t0"])], + *box_index(float(r["center_lat"]), float(r["center_lon"]))).astype(float) + for r in qual] + # R4's "box" for the degenerate patch is the storm whose |grad p| is the + # tier median, matching D-CZ-1's own choice for the same purpose. + boxes2d["R4_STORM"] = storm_boxes2d[int(np.argsort(storm_g)[len(storm_g) // 2])] + + samples = {k: v.ravel() for k, v in boxes2d.items()} + samples["R4_STORM"] = np.concatenate( + [b.ravel() for b in storm_boxes2d]).astype(float) + budget = min(len(v) for v in samples.values()) + for k in list(samples): + if len(samples[k]) > budget: + samples[k] = samples[k][rng.choice(len(samples[k]), budget, replace=False)] + assert len({len(v) for v in samples.values()}) == 1, "equal budget violated" + + def degenerate_patch_range(name): + """A real spatial corner -- n_i x n_i cells, n_i = max(1, side//8) -- + matching D-CZ-1's `p[si,sj][:n_i,:n_i]` construction exactly.""" + b2 = boxes2d[name] + n_i = max(1, b2.shape[0] // 8) + patch = b2[:n_i, :n_i] + return float(patch.min()), float(patch.max()) + + REG = list(samples) + + # ---- C2 / C3 / C4 / C6 : the transfer matrix ---- + matrix = {} + for T in REG: + truth = samples[T] + matrix[T] = {} + for D in REG: # CAL-ABS: a REAL donor matrix + dec, idx = encode_decode(truth, samples[D].min(), samples[D].max()) + matrix[T][f"CAL-ABS|{D}"] = cell_metrics(dec, idx, truth) + for D in REG: # dynamic arms: donor is IGNORED by + dec, idx = rank_codec(truth) # construction — that is the test + matrix[T][f"CAL-RANK|{D}"] = cell_metrics(dec, idx, truth) + dec, idx = fisherz_codec(truth) + matrix[T][f"CAL-FISHERZ|{D}"] = cell_metrics(dec, idx, truth) + perm = rng.permutation(N_LEVELS) # controls, on the diagonal + dec, idx = encode_decode(truth, truth.min(), truth.max(), decode_table=perm) + matrix[T]["CAL-SHUFFLE|self"] = cell_metrics(dec, idx, truth) + plo, phi = degenerate_patch_range(T) + dec, idx = encode_decode(truth, plo, phi) + matrix[T]["GEO-DEGENERATE|self"] = cell_metrics(dec, idx, truth) + + def L(arm, D, T): + return matrix[T][f"{arm}|{T}"]["rho"] - matrix[T][f"{arm}|{D}"]["rho"] + + def undefined_donors(T): + """Donors under which CAL-ABS's decode of T is fully saturated to one + level -- rho is undefined there, not merely small. A first-class + outcome, not a missing measurement: it means the donor's range is + so narrow relative to the target that structure is destroyed + entirely, the extreme end of what L is built to detect.""" + return [D for D in REG if np.isnan(matrix[T][f"CAL-ABS|{D}"]["rho"])] + + c2 = {"dynamic_rows_flat": {}, "cal_abs_can_differ": {}, + "cal_abs_undefined_donors": {T: undefined_donors(T) for T in REG}} + for arm in ("CAL-RANK", "CAL-FISHERZ"): + # dynamic arms have no donor at all, so L is 0 vs every D by + # construction -- never nan; a plain max is correct here. + c2["dynamic_rows_flat"][arm] = { + T: max(abs(L(arm, D, T)) for D in REG) for T in REG} + for T in REG: + defined = [abs(L("CAL-ABS", D, T)) for D in REG if D not in undefined_donors(T)] + c2["cal_abs_can_differ"][T] = max(defined) if defined else float("nan") + c2["half_i_dynamic_L_is_exactly_zero"] = bool(all( + v == 0.0 for a in c2["dynamic_rows_flat"].values() for v in a.values())) + c2["half_ii_cal_abs_nonzero_somewhere"] = bool( + np.nanmax(list(c2["cal_abs_can_differ"].values())) > 0) + + def l_bar(T): + """Mean off-diagonal transfer loss over DEFINED donors, with the + undefined (fully-saturated) donors counted separately rather than + silently dropped or silently nan-poisoning the mean.""" + vals = [L("CAL-ABS", D, T) for D in REG if D != T + and not np.isnan(matrix[T][f"CAL-ABS|{D}"]["rho"])] + return (float(np.mean(vals)) if vals else float("nan")), len(vals) + + c3 = {} + for T in REG: + lb, n_defined = l_bar(T) + c3[T] = {"L_bar": lb, "n_defined_offdiag": n_defined, + "n_undefined_offdiag": len(REG) - 1 - n_defined, + "occupancy_mean_offdiag": float(np.mean( + [matrix[T][f"CAL-ABS|{D}"]["occupancy"] for D in REG if D != T])), + "saturation_mean_offdiag": float(np.mean( + [matrix[T][f"CAL-ABS|{D}"]["saturation"] for D in REG if D != T]))} + # C3 as pre-registered compares L_bar(R4) to L_bar(R1). Both must be + # DEFINED for the comparison to mean what the bar says; report which + # case obtains rather than silently computing a NaN-laundered verdict. + c3_r1_r4_comparable = bool(np.isfinite(c3["R1_CALM"]["L_bar"]) + and np.isfinite(c3["R4_STORM"]["L_bar"])) + c3_holds = (bool(c3["R4_STORM"]["L_bar"] < c3["R1_CALM"]["L_bar"]) + if c3_r1_r4_comparable else None) + + c4 = {} + for T in REG: + d_rmse = (matrix[T][f"CAL-RANK|{T}"]["rmse_pa"] + - matrix[T][f"CAL-ABS|{T}"]["rmse_pa"]) + weak = float(np.mean([matrix[T][f"CAL-ABS|{D}"]["rmse_pa"] + for D in REG if D != T])) - matrix[T][f"CAL-RANK|{T}"]["rmse_pa"] + c4[T] = {"delta_rmse_rank_minus_absdiag": d_rmse, + "rho_floor_check_min": min(matrix[T][f"CAL-{a}|{T}"]["rho"] + for a in ("ABS", "RANK", "FISHERZ")), + "weak_form_absforeign_minus_rank_rmse": weak} + c4_flip = bool(c4["R1_CALM"]["delta_rmse_rank_minus_absdiag"] > 0 + and c4["R2_OCEAN"]["delta_rmse_rank_minus_absdiag"] > 0 + and c4["R4_STORM"]["delta_rmse_rank_minus_absdiag"] < 0) + + c0 = {} + for T in REG: + real = min(matrix[T][f"CAL-{a}|{T}"]["rho"] for a in ("ABS", "RANK", "FISHERZ")) + c0[T] = {c: bool(matrix[T][f"{c}|self"]["rho"] < real) + for c in ("CAL-SHUFFLE", "GEO-DEGENERATE")} + + result = { + "probe": "substrate_comfort_d_cz_2_7", "seed": SEED, + "timesteps": TIMESTEPS, "budget_cells_per_regime": int(budget), + "units": {"grad_p": "Pa/cell (flat, no cos-lat — the D-CZ-0 identified " + "definition)", "rmse": "Pa", "bias": "Pa", + "decay_length": "grid cells", "L": "dimensionless (rho difference)"}, + "C0_controls": c0, + "C1_ladder": {"per_timestep": c1, "holds_all": bool(c1_holds), + "raw": ladder}, + "C1b_separation": {"per_timestep": c1b, "bar": 3.0, + "holds_all": bool(all(v >= 3.0 for v in c1b.values()))}, + "C1c_structure": {"per_regime": c1c, "verdict": c1c_verdict}, + "C2_degenerate_row": c2, + "C3_transfer_loss": {"per_regime": c3, + "r1_r4_both_defined": c3_r1_r4_comparable, + "holds_r4_lt_r1": c3_holds}, + "C4_crossover": {"per_regime": c4, "sign_flip": bool(c4_flip)}, + "C6_matrix": matrix, + "c5_structural_blocker": ( + "C5 (GEO-GOLDEN-LO vs GEO-GOLDEN-HI) is NOT RUN, and the reason is " + "structural rather than budgetary. The golden index floor requires " + "N >= F(17)^2 = 2,550,409 lattice points before the emergent " + "parastichy stride behaves like phi at all. A 16-degree box at 0.25 " + "degree resolution contains 65 x 65 = 4,225 cells -- three orders of " + "magnitude below the floor. GEO-GOLDEN-HI therefore CANNOT be " + "constructed at box scale on this grid, so the comparison the bar " + "asks for has no admissible high arm. Reported rather than faked " + "with an interpolated sub-grid lattice, which would sample the " + "interpolator and not the field."), + } + OUT.write_text(json.dumps(result, indent=2)) + TAG.write_text("DONE d-cz-2..7\n") + + print(f"budget/regime = {budget} cells\n") + print("C1c STRUCTURE (runs first — a null here VOIDS the reading):") + for k, v in c1c.items(): + print(f" {k:10s} decay={v['decay_length_cells']:5.2f} cells " + f"gini={v['gini_gradp']:.4f} tail={v['tail_ratio_gradp']:.3f}") + print(f" R4/R1 ratios: {c1c_verdict}") + print(f"\nC1 ladder holds all timesteps: {c1_holds}") + print(f"C1b separation per timestep: " + f"{ {k: round(v,2) for k,v in c1b.items()} }") + print(f"\nC2 half(i) dynamic L==0 exactly: {c2['half_i_dynamic_L_is_exactly_zero']}") + print(f"C2 half(ii) CAL-ABS non-degenerate: {c2['half_ii_cal_abs_nonzero_somewhere']}" + f" max|L| per regime: { {k: round(v,4) for k,v in c2['cal_abs_can_differ'].items()} }") + print("\nC3 transfer loss L_bar (lower = travels better; " + "undefined = donor saturated the target to one level):") + for k, v in c3.items(): + lb = f"{v['L_bar']:+.5f}" if np.isfinite(v['L_bar']) else "UNDEFINED" + print(f" {k:10s} L_bar={lb:>10s} defined={v['n_defined_offdiag']}/" + f"{v['n_defined_offdiag']+v['n_undefined_offdiag']} " + f"occ={v['occupancy_mean_offdiag']:.3f} sat={v['saturation_mean_offdiag']:.3f}") + print(f" C3 R1/R4 both defined: {c3_r1_r4_comparable} " + f"holds (R4 < R1): {c3_holds}") + print("\nC4 crossover on RMSE (>0 = absolute wins, <0 = dynamic wins):") + for k, v in c4.items(): + print(f" {k:10s} delta={v['delta_rmse_rank_minus_absdiag']:+9.3f} Pa" + f" rho_floor={v['rho_floor_check_min']:.6f}") + print(f" C4 sign flip: {c4_flip}") + + +if __name__ == "__main__": + main() From 7a0ca11ea433dd2b27d192f808438b0b2fb0f01a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 06:47:45 +0000 Subject: [PATCH 4/8] CONFOUND: the regime ladder measured RANGE, not turbulence -- withdraw the "reversal" reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator asked whether storm modelling is sound given that vortices are not actively modelled. Checked, and it surfaces two things. FACT: substrate_comfort_d_cz_2_7.py loads ONLY mean_sea_level_pressure. No wind, no vorticity, no rotation in any bar. D-CZ-0/1 fetched 10m winds only to REPORT spd_sigma; nothing consumed them. R4 "STORM" is a scalar pressure-gradient regime, while the hypothesis was about "high velocity differences / turbulence" -- a dynamical property absent from every metric. CONFOUND, measured: Pearson(L, saturation) = +0.917, and Spearman(L-bar, regime's own value range) = +1.000, perfectly monotone. L is essentially how much of the target falls outside the donor's codebook range; R4's range is ~18x R1's. So C3's monotone rise largely RESTATES the width ordering the |grad p| ladder itself produced. Width is not the whole driver -- R3->R2 has a wider donor yet 0.949 saturation because the boxes sit at different absolute pressure levels -- so the real variable is coverage (width AND offset), which saturation captures. C4 is range-inflated identically: RMSE in Pa is not comparable across regimes differing 18x in range. Normalised as a ratio it reads 3.96 / 1.85 / 1.14 / 2.35 -- NOT monotone, and R1 is the extreme, not R4. SURVIVES: the hypothesis is not supported. No sign flip in either measure, normalised or not. WITHDRAWN: "cleanly reversed, monotonic" and "storms are less forgiving of bad calibration". As measured that says wide-range boxes are harder to cover with a foreign codebook, which is arithmetic, not meteorology. Why no gate caught it: C1c passed and was the right gate, but it measured the structure of the |grad p| FIELD, not rotational structure -- and the confound lives between the ladder's own discriminator and the codec's coverage. A gate cannot catch a confound inside the variable it was told to trust. The operator's question was the instrument; nothing in the apparatus was positioned to ask it. Propagated: plan header (the "monotonic reversal" headline replaced), §7.9 added with the full measurement, STATUS_BOARD D-CZ-4/D-CZ-5 qualified, new D-CZ-8 row (vorticity discriminator + range-normalised metric as the pre-condition for re-asking this as a turbulence question), EPIPHANIES E-THE-REGIME-LADDER-MEASURED-RANGE-NOT-TURBULENCE-1 (prepend, suffix-verified). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/EPIPHANIES.md | 70 ++++++++++++++ .claude/board/STATUS_BOARD.md | 5 +- .claude/plans/substrate-comfort-zones-v1.md | 101 ++++++++++++++++++-- 3 files changed, 165 insertions(+), 11 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index d97b448b4..64213ed32 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,73 @@ +## 2026-08-12 — E-THE-REGIME-LADDER-MEASURED-RANGE-NOT-TURBULENCE-1 + +**Status:** FINDING `[G]` — measured, same run, found by an operator +question rather than by any gate. Qualifies +`E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND-REVERSED-1`, written one commit +earlier, whose *stronger* half it withdraws. + +**The operator asked: do we have a storm-modelling problem, since we do +not actively model vortices?** Checked, and the answer is yes, twice over. + +**First, the plain fact.** `substrate_comfort_d_cz_2_7.py` loads **only** +`mean_sea_level_pressure`. No wind, no vorticity, no rotation enters any +bar. D-CZ-0/1 fetched 10 m winds solely to *report* `spd_sigma`; nothing +consumed them. R4 "STORM" is a **scalar pressure-gradient** regime, and +the hypothesis it was built to test was about *"high velocity differences +/ turbulence"* — a dynamical property that appears nowhere in the +measurement. + +**Second, the confound that follows.** Chasing the question produced two +measurements that qualify the previous entry: + +| relationship | statistic | +|---|---| +| `L` vs the cell's `saturation` | Pearson **+0.917** (n=8) | +| `L̄[T]` vs the regime's own value range | **Spearman +1.000 — perfect** | + +`L` is essentially *how much of the target falls outside the donor's +codebook range*. A regime's own range determines how hard it is to cover +— R4's is ~**18×** R1's. So C3's monotone rise **restates the width +ordering**, which the `|∇p|` ladder itself produced (deeper low ⇒ steeper +gradient ⇒ wider range). Width is not the whole story — `R3 → R2` has a +*wider* donor yet 0.949 saturation, because the boxes sit at different +absolute pressure levels — so the real driver is **coverage** (width AND +offset), which `saturation` captures directly. + +C4 is inflated the same way: `Δ` was amended to RMSE **in Pa**, and RMSE +scales with range, so an absolute-Pa margin cannot be compared across +regimes differing 18×. Normalised as a ratio it reads **3.96 / 1.85 / +1.14 / 2.35** — not monotone, and R1 is the extreme, not R4. + +**What survives / what is withdrawn:** + +- **SURVIVES:** the hypothesis is **NOT SUPPORTED**. No sign flip in + either measure, normalised or not; `CAL-ABS` wins its own diagonal in + all four regimes. Unaffected by the confound. +- **WITHDRAWN:** *"cleanly reversed, monotonic"* and *"storms are LESS + forgiving of bad calibration."* As measured that says **wide-range + boxes are harder to cover with a foreign codebook** — arithmetic. + +**Why no gate caught this.** C1c *did* pass, and it was the right gate to +have — but it measured the structure of the **`|∇p|` field** (decay +length, Gini, tail ratio), not rotational structure, and the confound is +between the ladder's own discriminator and the codebook's coverage, not +between two regimes. A gate cannot catch a confound that lives *inside* +the variable it was told to trust. **The operator's question was the +instrument here; nothing in the apparatus was positioned to ask it.** + +**The transferable rule.** *When a regime axis is built from a scalar +derived from the same field the codec quantises, the axis and the codec's +difficulty are not independent.* Before reading a cross-regime encoding +result as physics, check whether the regime discriminator predicts the +encoding's own failure mode — here, `|∇p|` predicts range, and range +predicts saturation, and saturation IS `L`. A dynamical hypothesis needs +a **dynamical** discriminator (ζ = ∂v/∂x − ∂u/∂y, Okubo–Weiss) and a +**range-normalised** transfer metric, or coverage will masquerade as the +finding every time. + +**Filed as D-CZ-8**, the pre-condition for re-asking C3/C4 as a turbulence +question rather than a width question. + ## 2026-08-12 — E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND-REVERSED-1 **Status:** FINDING `[G]` — measured, both pre-registered bars, one run. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 7afe6d922..92c73e494 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -39,9 +39,10 @@ already run and it corrected two regime definitions before any bar existed. | D-CZ-2b | **C1b constancy is measured** — `separation ≥ 3` | **DONE — PASS.** separation = **5.87–8.24** at all 3 timesteps | earns the phrase "held constant" | | D-CZ-2c | **C1c the suitability ASSUMPTION** | **DONE — PASS, distinguishable=True.** Decay/Gini/tail-ratio R4-vs-R1 ratios 0.88/0.71/**0.385** — all deviate ≥ 20 % from 1. **Licenses the cross-swap interpretation.** A construction bug in C0 was found and disable-verified fixed en route (§7.0) — `GEO-DEGENERATE` was built from a shuffled array slice, not a spatial patch | was a null-and-void gate; passed, so downstream stands | | D-CZ-3 | **C2 degenerate-row verification, both halves** | **DONE — PASS both halves.** Dynamic `L ≡ 0` exactly for every donor; `CAL-ABS` off-diagonal `L` ranges **0.011 (R1) → 0.947 (R4)**, demonstrably non-degenerate | the can-it-DIFFER gate; satisfied | -| D-CZ-4 | C3 **transfer loss** vs turbulence | **DONE — FAILS, and REVERSED.** `L̄[T]` = **0.011 → 0.309 → 0.671 → 0.690**, R1→R4 — a monotonic **increase**, not the required decrease. Storms are LESS forgiving of bad calibration on this data, not more (§7.3) | refutes "storms forgive bad calibration" | -| D-CZ-5 | **C4 the crossover** | **DONE — FAILS, no sign flip.** `Δ` (absolute − dynamic RMSE, own diagonal) = **+1.29 / +1.11 / +0.46 / +10.78 Pa** R1→R4 — absolute wins everywhere, by a GROWING margin. Weak form passes trivially as pre-registered (329–2508 Pa, "evidence of wiring, not merit") (§7.4) | **the operator's hypothesis, two-sided — REFUTED** | +| D-CZ-4 | C3 **transfer loss** vs turbulence | **DONE — FAILS. ⚠ But the "reversal" reading is WITHDRAWN (§7.9).** `L̄` = 0.011 → 0.309 → 0.671 → 0.690 is **rank-correlated ρ = 1.000 with each regime's own value range**, and `L` tracks `saturation` at Pearson **+0.917** — it largely restates *wide boxes are hard to cover*, arithmetic not meteorology. What stands: the hypothesis is not supported | hypothesis unsupported; the causal reading is confounded | +| D-CZ-5 | **C4 the crossover** | **DONE — FAILS, no sign flip** (that part is solid). ⚠ The "+10.78 Pa growing margin" is **range-inflated** — RMSE in Pa is not comparable across regimes differing 18× in range; normalised as a ratio it reads 3.96 / 1.85 / 1.14 / **2.35**, NOT monotone, R1 extreme (§7.9). Weak form passes trivially as pre-registered | hypothesis **not supported**; margin trend withdrawn | | D-CZ-6 | C5 geometry floor | **NOT RUN — structural blocker.** Box holds 4225 cells; the golden index floor needs N ≥ 2 550 409. `GEO-GOLDEN-HI` cannot be constructed at box scale on this grid (§7.5) | reported, not faked with an interpolated lattice | +| D-CZ-8 | **NEW — vorticity regime + range-matched transfer metric.** Every bar so far runs on a SCALAR pressure field; no wind, no rotation enters any metric, so "turbulence" was never operationalised dynamically (§7.9). Needs ζ = ∂v/∂x − ∂u/∂y (or Okubo–Weiss) as the discriminator AND a range-normalised `L` so coverage cannot masquerade as the finding | pre-condition for re-asking C3/C4 as a turbulence question | | D-CZ-7 | C6 the transfer matrix (the deliverable) | **DONE.** Full donor×target table, all 5 arms × 4 regimes, every cell raw in `substrate_comfort_d_cz_2_7.json` → `C6_matrix` (§7.6) | comfort read off the diagonal, travel-cost off the off-diagonal | > **2026-08-12 — rows re-cut, not restated.** The operator ruled that the diff --git a/.claude/plans/substrate-comfort-zones-v1.md b/.claude/plans/substrate-comfort-zones-v1.md index 241818cb8..56ba9c44a 100644 --- a/.claude/plans/substrate-comfort-zones-v1.md +++ b/.claude/plans/substrate-comfort-zones-v1.md @@ -6,15 +6,26 @@ > rule). No verdict-tier claim may be promoted out of this plan without an > independent adversarial spec audit. > -> **⚠ HEADLINE, READ FIRST — see §7 for the full run.** Both pre-registered -> measures of the hypothesis below point the SAME direction, and it is the -> OPPOSITE of what was hypothesized: `C3`'s transfer loss `L̄` grows -> monotonically 0.011 → 0.309 → 0.671 → **0.690** from calm to storm (not -> shrinks); `C4`'s crossover never flips — absolute wins its own diagonal -> in every regime, by a MARGIN THAT GROWS from ~1 Pa to **10.78 Pa** in the -> storm regime, not shrinks or reverses. On this box, this variable -> (MSLP), these timesteps: **a well-calibrated absolute encoding does not -> lose its edge under turbulence — it gains one.** +> **⚠ HEADLINE, READ FIRST — see §7, and §7.9 for the confound.** +> **The hypothesis is NOT SUPPORTED:** there is no sign flip in either +> pre-registered measure, normalised or not — `CAL-ABS` wins its own +> diagonal in all four regimes. That much is solid. +> +> **But the stronger "cleanly reversed, monotonic" reading is WITHDRAWN** +> (§7.9, triggered by the operator asking whether storm modelling is sound +> given that vortices are not modelled at all). `C3`'s monotone `L̄` rise +> is **rank-correlated ρ = 1.000 with each regime's own value range**, and +> `L` tracks `saturation` at Pearson **+0.917** — so it largely restates +> *wide-range boxes are hard to cover with a foreign codebook*, which is +> arithmetic, not meteorology. `C4`'s "+10.78 Pa margin" is range-inflated +> for the same reason; normalised as a ratio it reads 3.96 / 1.85 / 1.14 / +> **2.35** — not monotone, and R1 is the extreme, not R4. +> +> **Scope, stated bluntly:** every bar here runs on a **scalar pressure +> field**. No wind, no vorticity, no rotation enters any metric. This plan +> answered *"absolute vs window-local encoding of a scalar field whose +> range varies by regime"* — a real question — but **not** the operator's +> original question about turbulence. > > **Operator framing (2026-08-12), three messages:** > 1. *hold different situations constant — over water vs flatland vs storm @@ -793,3 +804,75 @@ means the geometry axis (the *other* half of "good geometry vs badly calibrated") was never tested at all. The regime axis and the C0/C1/C1b/ C1c apparatus all hold; the specific calibration hypothesis they were built to test does not. + + +### §7.9 ⚠ CONFOUND FOUND POST-RUN — C3's monotonicity is largely RANGE, not turbulence + +**Trigger (operator, 2026-08-12):** *do we have a problem with the storm +modelling, since we do not actively model vortices?* The question is +correct, and chasing it surfaced a confound that qualifies §7.3 and §7.4. + +**Fact first:** `substrate_comfort_d_cz_2_7.py` loads **only** +`mean_sea_level_pressure`. No wind, no vorticity, no rotation. D-CZ-0/1 +loaded 10 m winds solely to *report* `spd_sigma`; no bar ever consumed +them. R4 "STORM" is therefore characterised by a **scalar pressure-gradient +magnitude**, not by the rotating structure that makes a storm a storm. + +**The confound that follows, measured:** + +| relationship | statistic | +|---|---| +| `L` vs the cell's `saturation` | Pearson **+0.917**, Spearman **+0.833** (n=8) | +| `L̄[T]` vs the regime's OWN value range | **Spearman +1.000 — perfectly monotone** | + +`L` is essentially a function of **how much of the target's distribution +falls outside the donor's codebook range**. A regime's own range is what +makes it hard to cover: R4's implied range is ~**18×** R1's (≈7075 Pa vs +≈386 Pa). So `L̄` rising monotonically R1→R4 substantially **restates the +regimes' width ordering**, which the `|∇p|` ladder itself produced (a +deeper low means a steeper gradient *and* a wider box range). + +Width alone is not the whole mechanism — `R3 → R2` has a *wider* donor +(log₂ ratio +1.32) yet saturation 0.949, because the two boxes sit at +different absolute pressure levels. The true driver is **coverage** +(width **and** offset), which `saturation` captures directly and which the ++0.917 correlation measures. + +**C4 is range-inflated the same way.** `Δ` was amended to RMSE **in Pa** +(§6.4), and RMSE scales with the field's range, so an absolute-Pa margin is +not comparable across regimes differing 18× in range. Normalised as a +ratio, C4 reads **3.96 / 1.85 / 1.14 / 2.35** (R1→R4) — **not monotone**, +and R1, not R4, is where absolute wins by the largest factor. + +**What survives, and what does not:** + +- **SURVIVES — the hypothesis is NOT SUPPORTED.** There is no sign flip in + either metric, normalised or not; `CAL-ABS` wins its own diagonal in all + four regimes. That conclusion is unaffected by the confound. +- **DOES NOT SURVIVE — "cleanly reversed, monotonic."** §7.3's monotone + `L̄` increase is confounded with range (ρ = 1.000), and §7.4's "margin + grows to 10.78 Pa" is range-inflated. **"Storms are LESS forgiving of bad + calibration" is withdrawn as a physical claim** — as measured it says + *wide-range boxes are harder to cover with a foreign codebook*, which is + arithmetic. +- **C1c still passes** and still licenses the design — but note what it + measured: structure of the **`|∇p|` field**, not rotational structure. + It cannot have caught this, because the confound is between the + ladder's own discriminator and the codebook's coverage, not between two + regimes. + +**The deeper gap this exposes.** The hypothesis was about *"high velocity +differences / turbulence"* — a **dynamical** property. Every bar in this +plan is computed on a **scalar pressure field**. A vortex is not +represented anywhere: not in the regime definition, not in the codebook, +not in any metric. So this plan has tested *"badly-calibrated absolute vs +window-local encoding of a scalar field whose range varies by regime"* — +which is a real question, and answered — but it is **not** the operator's +original question about turbulence, and should stop being described as if +it were. + +**Consequence for the next run:** a genuine turbulence regime needs a +**rotational** discriminator (relative vorticity ζ = ∂v/∂x − ∂u/∂y, or +Okubo–Weiss), and a range-matched or range-normalised transfer metric so +coverage cannot masquerade as the finding. Both are pre-conditions for +re-asking C3/C4 as a turbulence question rather than a width question. From 4feb367b22af143577e55efdfe75eec2194844fe Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 09:18:05 +0000 Subject: [PATCH 5/8] PLAN: weather-soa-bake-v1 -- the Zarr->NodeRow bake the arc never had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The weather arc ran entirely as Python over a Zarr file -- one variable, four hand-picked 16-degree boxes, three hand-typed timesteps -- and the reason is structural, not a choice: `crates/weather-poc` does not exist, and `weather-substrate-poc-v2.md` names it, declares "New repositories required: ZERO", and then specifies NO bake step at all. The plan closes that. One cell = one node; one timestep = one Lance version. HEEL = the 16-degree tile, HIP = position within it, TWIG dormant-reserved -- so the arc's hand-picked boxes become HEEL-prefix range scans, and a lat/lon grid is the literal-x/y case OGAR's 256x256 centroid-tile cascade sanctions (5-6 of 12 quaternary levels per axis; 4^5 >= 721, 4^6 >= 1440). Value lane: L4 6x(8:8) palette256, one byte per field, pairs as physical couples, with a STATED deviation from L4's Fisher-z default to the linear helix RollingFloor -- licensed by le-contract §3's own monotone-bounded demarcation and by the measurement that Fisher-z burns 228/256 buckets on ERA5 temperature. Five corrections to the framing it was commissioned under, each verified: - Capacity: "32 facets/cell" overstates the usable budget by ~78%. Two slots are key+edges and VALUE_TENANTS carves the slab contiguously; the committed assertion value_tenants_contiguous_within_slab pins the Full carve at 188 of 480 B, so the free budget is 292 B = 18 facets = 216 payload bytes, not 384. The headline survives -- all 122 ERA5 fields fit at 1 B/field, one cell is one node -- but at 2 B/field they do NOT fit as 4+12 facets. - D-WXA-5 is not on STATUS_BOARD, and neither is any D-WXA-*/D-WXB-*/D-WXC-* row (grep -c = 0, verified). That ladder's board hygiene was never discharged -- a plausible mechanism for a whole arc running past an unmet gate. - D-WXA-5's rho >= 0.98 bar is probably unable to fail: D-CZ-1 §6.4 measured real-arm rho spread at 3e-6..4.7e-5. Re-specified in three parts, including a can-it-DIFFER resolution ladder that must run BEFORE any verdict. - The ~8.1 s/timestep figure is both a serial extrapolation from ONE measured cycle (D-KIA-A2 unbuilt) and a category error -- a bake is a columnar write, not 16 mailbox seal cycles. D-WXS-11 measures it instead of refining it. - The full grid does NOT dissolve the §7.9 range confound. L~saturation +0.917 and Lbar~range +1.000 is arithmetic that 1.04 M cells do not repeal. What it buys is coverage-matched donor selection, a zeta-band regime definition, and a regime axis that is not n = 4. Adds one falsifier the arc never had: differencing amplifies quantisation error, so quantised-zeta vs true-zeta is reported per zeta-magnitude decile with a laminar stay-silent twin, and a KILL that makes zeta its own baked lane. Board hygiene in the same commit: STATUS_BOARD gains the D-WXS block (13 rows, D-WXS-0 Blocked on the OGAR-side classid mint) and INTEGRATION_PLANS gains its PREPEND entry. Both files verified pure-prepend (suffix-preserved). Doc-only. No code, no crate, no cargo run. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/INTEGRATION_PLANS.md | 79 +++ .claude/board/STATUS_BOARD.md | 43 ++ .claude/plans/weather-soa-bake-v1.md | 759 +++++++++++++++++++++++++++ 3 files changed, 881 insertions(+) create mode 100644 .claude/plans/weather-soa-bake-v1.md diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index de6ac45c8..60705ceb5 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,82 @@ +## 2026-08-13 — weather-soa-bake-v1 (PLAN; the missing Zarr→NodeRow bake) + +Plan: `.claude/plans/weather-soa-bake-v1.md`. **Does not supersede +`weather-substrate-poc-v2.md`** — §9 grades it part by part. The gap it closes: +poc-v2 names `crates/weather-poc` and declares "New repositories required: +ZERO", but the crate does not exist and the plan describes **no bake step at +all** — which is why the entire weather arc ran as Python probes over a Zarr +file, one variable, four hand-picked 16° boxes, three hand-typed timesteps. + +**The bake.** One cell = one node, one timestep = one Lance version. The key: +HEEL = `(lat>>6, lon>>6)` = the 16°×16° tile, HIP = position within it, TWIG +dormant-reserved — so a lat/lon grid is the literal-x/y binding OGAR's +256×256-centroid-tile cascade sanctions, and **the arc's hand-picked 16° boxes +become HEEL-prefix range scans**. The grid needs 5–6 of the 12 available +quaternary levels per axis (4⁵ ≥ 721, 4⁶ ≥ 1440), leaving room for 0.05° +without touching the stride. Three wrinkles stated, not designed around: both +axes tile **raggedly** (721 = 11×64+17; 1440/64 = 22.5), longitude is cyclic so +a seam-crossing box is **two** prefix ranges, and `MailboxId ≠ NiblePath` in +code so nothing leans on it. + +**The value.** L4 `6×(8:8)` palette256², one byte per ERA5 field, pairs being +physical couples (`(u:v)`, `(T:q)`) so a wind-vector distance is ONE 256×256 LUT +read. Argued against the whole le-contract §3 catalogue: L1–L3 are relational, +L5 kept only as the 3-D-wind alternative, L6 is the Odoo factoring, **L7 is +location and location is the KEY**, L8 is retired for this data (§12.12 RETIRED +the `Pair48` mint), G1–G3 are forbidden to new classes. **Stated deviation:** +L4's canonical Fisher-z read is replaced by the linear `helix::RollingFloor` — +licensed by le-contract §3's own monotone-bounded-continuous-field demarcation +and by §12.1's measurement (Fisher-z burns 228/256 buckets on ERA5 temperature). +Circular variables never ride an L1 byte (359°/1° at max L1 distance); wind +rides as `(u,v)`, and `wind_speed` is baked separately because §6.5 measured the +Jensen gap **is** gustiness (mean ratio 1.115, max gap 14.37 m/s). + +**Capacity, corrected.** "32 facets/cell" overstates the usable budget: 2 slots +are key+edges and `VALUE_TENANTS` carves the slab contiguously through offset +220 (committed assertion: **188 of 480 B**), so the free budget is **292 B = 18 +facets = 216 payload bytes** (discriminant 15 reserved for `BoardAggregates`; +weather takes 16, offset DERIVED). All 122 fields still fit at 1 B/field — +**one cell is one node** — but at 2 B/field they do NOT fit as 4+12 facets +(336 > 292). The 14 statics get their own classid and ONE version (baking them +per timestep would rewrite ~1.3 PB of constants). W1 bakes a minimal justified +set, not all 122, so the layout does not presuppose the fidelity gate's outcome. + +**Time.** `E-MARKOV-TEMPORAL-STREAM-1` applied literally: a timestep is a Lance +version, a time series is a version-range read (`QueryReference::at(v, rung)` + +`deinterlace`), the 19-storm set is a version set. Zero versioning code — the +`LanceCycleWriter` / `VersionedGraph` surface is consumed. Open risk, +pre-registered: 92,044 versions on one dataset is unmeasured (`D-WXS-6`, with a +KILL). + +**Re-homing.** `D-WXA-5` moves onto the substrate AND is re-specified: poc-v2's +ρ ≥ 0.98 is at serious risk of being vacuous because D-CZ-1 §6.4 measured +real-arm ρ spread at 3e-6…4.7e-5 — so the bar becomes (a) ρ ≥ 0.9996 (where a +real pair genuinely failed at 0.999556), (b) the shuffled control must fall +below 0.98, and (c) a 16/64/256-level ladder must be monotone BEFORE any verdict +(the can-it-DIFFER half). `D-CZ-8` moves on as ζ from a neighbour-key stencil — +with a falsifier the arc never had: **differencing amplifies quantisation +error**, so quantised-ζ vs true-ζ is reported per ζ-magnitude decile with a +laminar stay-silent twin. On §7.9: the full grid **does NOT dissolve the range +confound** (`L`~saturation +0.917, `L̄`~range +1.000 is arithmetic that 1.04 M +cells do not repeal); what it buys is (1) coverage-matched donor selection by +construction, (2) "donor" meaning a ζ-regime rather than a range once the shared +floor makes variables commensurable, (3) a regime sample that is not n = 4. + +**Splits.** Judgment (layout, key, every bar, the `D-WXA-5` re-spec, the +confound treatment, the mint) stays on Opus/main. Grindwork is +one-file-per-Sonnet-worker (`key.rs` / `floor.rs` / `lane.rs` / `bake.rs` / +`statics.rs` / `stencil.rs` / `metrics.rs`, one test file per bar), guardrails §1 +pasted verbatim, tag-files only. Orchestrator-only: `lib.rs` mod lines, the crate +manifest, `canonical_node.rs`, every board file. Root `Cargo.toml` untouched — +`weather-poc` is workspace-EXCLUDED. SIMD only via `ndarray::simd::*`; ndarray +optional, feature-gated, GIT-sourced (never an optional `path` dep). No +lance-family bump. Doc-only. + +**Also recorded:** `weather-substrate-poc-v2.md`'s `D-WXA-*`/`D-WXB-*`/`D-WXC-*` +ladder has **no rows on `STATUS_BOARD.md`** (`grep -c WXA` = 0, verified) — its +board hygiene was never discharged, which is a plausible mechanism for how a +whole arc ran past an unmet gate. + ## 2026-08-12 — substrate-comfort-zones-v1 REVISED (§2 rebuilt: horse race → cross-swap) Same file, `.claude/plans/substrate-comfort-zones-v1.md`, still **ACTIVE**, diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 92c73e494..ca527a363 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,46 @@ +## weather-soa-bake-v1 — Zarr → NodeRow, the missing bake (PRE-REGISTERED 2026-08-13) + +Plan: `.claude/plans/weather-soa-bake-v1.md`. The arc's Python-over-Zarr phase +ran because the Zarr→`NodeRow` path was never specified — +`weather-substrate-poc-v2.md` names `crates/weather-poc` and describes **no bake +step**. Waves W0–W5; every bar pre-registered and committed BEFORE its run, each +with a control that can lose and a stay-silent twin. + +| D-id | Deliverable | Wave | Status | Feeds | +|---|---|---|---|---| +| D-WXS-0 | classid mint for the weather-cell + statics classes (OGAR-side; `0x0F = Geo` exists, appid/classview open) | 0 | **Blocked (operator/OGAR)** | the bake must REFUSE to write under `0x0000_0000` | +| D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | Queued | slot purity §2; bar B0 | +| D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | Queued | ends the chat-only-figure defect for the census | +| D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | Queued | a 16° box becomes a HEEL-prefix scan; bar B1 | +| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | Queued | bar B2 | +| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | +| D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | +| D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions | +| D-WXS-7 | **D-WXA-5 re-homed and RE-SPECIFIED** — ρ(code_dist, field_dist) via `jc::reliability::spearman` over whole-grid pairs. (a) ρ ≥ 0.9996 (the bar a real pair FAILED at 0.999556); (b) shuffled-codebook control < 0.98 (measured losable at 0.003–0.159); (c) 16/64/256-level ladder must be MONOTONE before any verdict | 3 | Queued | ⚠ poc-v2's ρ ≥ 0.98 is at risk of being vacuous — D-CZ-1 §6.4 measured real-arm ρ spread 3e-6…4.7e-5 | +| D-WXS-8 | cross-variable comparability at grid scale (≥4 variables, ≥2 units, ≥3 seasons); per-variable-floor control must LOSE | 3 | Queued | P2 (0.9997 vs 0.857–0.875) promoted from [H] or refuted | +| D-WXS-9 | ζ = ∂v/∂x − ∂u/∂y as a substrate read (neighbour-key stencil) + the differencing-amplifies-quantisation falsifier, reported per ζ-magnitude decile | 4 | Queued | bar B8; KILL ⇒ ζ becomes its own baked lane | +| D-WXS-10 | **D-CZ-8 re-homed** — ζ-percentile regime bands over the WHOLE grid + coverage-matched donors + range-normalised `L`; shuffled-ζ control; same-band stay-silent twin | 4 | Queued | the grid buys CONTROL of the §7.9 confound, NOT its dissolution; KILL if `L̄`~range ρ≈1.0 survives matching | +| D-WXS-11 | MEASURE a full-grid bake wall time; state the ~8.1 s serial extrapolation as the prior and confirm/correct it in the same artifact | 5 | Queued | replaces an extrapolation from ONE cycle (514 ms / 65,536 owners); D-KIA-A2 still unbuilt | +| D-WXS-12 | jc ↔ ndarray reliability agreement (poc-v2 D-WXB-4, carried verbatim; jc is the authority, degenerate input reported not folded) | 5 | Queued | every Phase-A/C number is computed with jc | + +**Capacity, corrected in the plan (§0.4).** "32 facets/cell" overstates the +usable budget: 2 slots are key+edges and `VALUE_TENANTS` carves the slab +contiguously — the committed assertion +(`canonical_node.rs` `value_tenants_contiguous_within_slab`) pins the current +Full carve at **188 of 480 B**, so the free budget is **292 B = 18 facets = +216 payload bytes**, not 384. All 122 fields still fit at 1 B/field — **one +cell is one node** — but at 2 B/field they do **not** fit as 4+12 facets. + +> **Board-hygiene note (2026-08-13):** `weather-substrate-poc-v2.md`'s +> `D-WXA-*` / `D-WXB-*` / `D-WXC-*` ladder has **no rows on this board** +> (`grep -c WXA` = 0, verified). None of those deliverables was ever built. +> The rows above do not supersede them; the poc-v2 ladder should be landed as +> its own block, marked NEVER BUILT, so the gap is visible rather than inferred. + +> **C5 note:** the global grid does **not** unblock `GEO-GOLDEN-HI`. The golden +> index floor needs N ≥ F(17)² = 2,550,409; the grid has 1,038,240. Short by +> ~2.5×, not by three orders of magnitude — still not constructible. + ## SUBSTRATE_FORMULA_MATRIX — the arc's rated inventory (2026-08-12) Document: `probes/weather-p1/SUBSTRATE_FORMULA_MATRIX.md`. Not a plan and not diff --git a/.claude/plans/weather-soa-bake-v1.md b/.claude/plans/weather-soa-bake-v1.md new file mode 100644 index 000000000..cb2d5dd29 --- /dev/null +++ b/.claude/plans/weather-soa-bake-v1.md @@ -0,0 +1,759 @@ +# weather-soa-bake-v1 — Zarr → NodeRow, the missing bake, and the re-homing of the open weather work onto the substrate + +> **Status:** PLAN (2026-08-13). Doc-only. No code shipped, no board file written by this +> plan (the orchestrator is the sole writer of `.claude/board/*`; proposed rows are in §11). +> **Does not supersede** `weather-substrate-poc-v2.md` — §9 states part-by-part what survives, +> what is superseded by having the data on the substrate, and what was never built. +> +> **The one-sentence reason this plan exists:** an extended weather R&D arc ran entirely as +> Python over a Zarr file because **the Zarr→`NodeRow` path does not exist and no plan ever +> specified one** — `weather-substrate-poc-v2.md` names `crates/weather-poc`, declares "New +> repositories required: ZERO", and then describes encoder arms with **no bake step at all**. + +--- + +## §0 Corrections to the framing this plan was commissioned under + +Stated first, per the workspace rule that a plan which designs around a wrong premise is worse +than one that names it. + +**§0.1 — `D-WXA-5` is NOT on `STATUS_BOARD.md`. Neither is any `D-WXA-*` / `D-WXB-*` / +`D-WXC-*` row.** `grep -c "WXA" .claude/board/STATUS_BOARD.md` = **0** (verified this session, +and re-verified by the orchestrator before landing). The D-ids exist only inside +`.claude/plans/weather-substrate-poc-v2.md` §2–§4 and in the `INTEGRATION_PLANS.md` prose entry +dated 2026-08-10. The board-hygiene rule (`CLAUDE.md` § Mandatory Board-Hygiene Rule, row "A +new D-id / deliverable") was **not discharged** for that plan. Consequence: the weather POC's +deliverable ladder has never been visible on the deliverable dashboard, which is a plausible +mechanism for how a whole arc ran past it in Python without anyone tripping over an unmet gate. + +**§0.2 — the label "the C2 gate" does not match the artifacts.** In +`weather-substrate-poc-v2.md` §2, `D-WXA-5` is *"the gate"* (PASS: ρ ≥ 0.98 for ≥1 arm **AND** +the shuffled-codebook control FAILS). `C2` is a bar in a *different* plan +(`substrate-comfort-zones-v1.md` §3 — the degenerate-row verification). They are unrelated +bars. This plan uses `D-WXA-5` and never the C2 label. + +**§0.3 — `D-WXA-5` as pre-registered is at serious risk of being VACUOUS, and that is the +single most load-bearing thing this plan has to say about it.** `substrate-comfort-zones-v1.md` +§6.4 **measured** that Spearman ρ *saturates* on smooth pressure fields at 256 levels: the +spread between the two real arms was **3×10⁻⁶ … 4.7×10⁻⁵**, and the plan's own C4 bar +*"could not have fired."* `D-WXA-5`'s bar is ρ ≥ 0.98 between code-distance and +field-distance — four orders of magnitude below where real arms actually land. **Every real +arm will pass it.** A bake-off in which every arm passes has measured nothing, which is what +`D-WXA-5`'s own anti-vacuity paragraph says. §5 below re-specifies the bar. This is not a +licence to weaken the gate; it is the `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1` rule applied +to the gate itself, using a number already committed to an artifact. + +**§0.4 — "32 × (4+12) facets per cell" and "33.2 M facets per timestep" are arithmetically +right and semantically misleading.** `GUIDS_PER_NODE = 32` is const-asserted +(`canonical_node.rs:805-808`, `512 / 16`), and 1,038,240 × 32 = 33,223,680. But: **2 slots are +`key` and `edges`**, and the 480-byte value slab is **not free space** — `VALUE_TENANTS` carves +it *contiguously, gap-free, compile-asserted*. The committed assertion is exact +(`canonical_node.rs:2292-2307`, `value_tenants_contiguous_within_slab`): **the current Full +carve uses 188 of 480 B** (kanban×Rubicon 8 + autopoiesis triangle 3×12 = 36 + TEKAMOLO facet +16 + CausalWitness facet 16), ending at row offset **220**. Discriminant **15 is reserved for +`BoardAggregates`** (`canonical_node.rs:1027-1033`); a new tenant takes 16. + +**Free budget for a weather lane today = 480 − 188 = 512 − 220 = 292 bytes** = 18 whole +16-byte facets (+4 spare bytes) = **216 bytes of facet payload**. §2 does the capacity +arithmetic against that number, not against 384. + +**§0.5 — the ~8.1 s/timestep figure is a serial extrapolation from ONE measured cycle, and it +is also a category error.** The measurement (`AGENT_LOG.md` 2026-08-05, PROBE-IGNITION-64K) +is: **65,536 real `MailboxSoA<4>` owners**, build 8.4 s, c1 cast 225 ms, **seal+apply 514 ms**, +c2 rest 73 ms. 1,038,240 / 65,536 = 15.84 ⇒ 16 × 514 ms = **8.22 s** — arithmetic `[G]`, +*conclusion* **CONJECTURE**, for three independent reasons: + +1. It assumes seal+apply is **linear in owner count**. `D-KIA-A2` — the pre-registered + parallelism falsifier (median-of-5, ≥2× at ≥4k owners, ±10 % stay-silent twin) — is + **`Queued` and unbuilt**. Nothing has been measured about scaling. +2. The measured run bought scale on the **owners** axis only (`MailboxSoA<4>`, **one populated + row per owner**). Weather's shape is the *rows-per-owner* axis, which that probe explicitly + did not exercise. +3. **A bake is not a mailbox cycle.** Writing 1,038,240 rows into a Lance version is a + columnar write; seal+apply is the cognition loop *over* rows that already exist. Quoting + 514 ms/cycle as a bake cost conflates two different operations. §4's `D-WXS-11` replaces + the extrapolation with a measurement rather than refining it. + +Everywhere below, the 8.1 s figure appears only as **CONJECTURE (serial extrapolation from one +measured cycle; `D-KIA-A2` unbuilt)**. + +**§0.6 — variable-census inputs used by this plan** (from a live `.zmetadata` read of +`1959-2022-6h-1440x721.zarr`, and **required to be committed as a re-runnable probe** — +`D-WXS-1a` — because a census asserted in prose is exactly the chat-only-figure defect this +arc has already catalogued three times): 17 time-varying surface + 7 pressure-level × 13 +levels = 91 + 14 static = **122 fields/cell**; time axis 92,044 six-hourly steps. Per +timestep: 1,038,240 × 122 = **126,665,280 values** landing in 1,038,240 × 512 B = +**531,578,880 B = 0.495 GiB** (arithmetic `[G]`). + +--- + +## §1 The key — does HEEL/HIP/TWIG address 721 × 1440 directly? + +**Verdict: YES, directly, with three named wrinkles. This is the literal-x/y case OGAR's +cascade doctrine explicitly sanctions, and it is the cleanest instance of prefix-routing in +the workspace.** + +### §1.1 The arithmetic + +`/home/user/OGAR/CLAUDE.md` § *Tier interpretation — 256×256 CENTROID TILE*: each tier's 16 +bits reads as a **256×256 tile** = two axes × one byte, nibble-interleaved; and the rigour +condition is **256 = 4⁴ — each codebook is a 4-level 4-ary centroid hierarchy**, so a byte's +nibbles are ancestry and `is_ancestor_of` = containment. Three tiers × 4 quaternary levels = +**12 quaternary levels per axis** = 4¹² = 16,777,216 addressable positions per axis. + +The grid needs 721 lats and 1440 lons. 4⁵ = 1024 ≥ 721; 4⁶ = 4096 ≥ 1440. **The 0.25° global +grid consumes 5–6 of the 12 available quaternary levels per axis** — leaving room for a 0.05° +grid (3600 × 7200) inside the same three tiers without touching the stride. + +**And the hierarchy condition holds *exactly*, not approximately.** OGAR's warning is +*"flat k-means-256 breaks this; hierarchical 4⁴ preserves it."* For a literal x/y grid the +"centroid codebook" is the **identity mipmap**: a coarse tile *contains* its fine cells by +construction. There is no codebook to train and nothing to certify — `is_ancestor_of` is exact +containment. That makes the weather grid the **strongest available test case** for the whole +prefix-routing claim, and §4's `D-WXS-2` turns it into a test rather than a remark. + +### §1.2 The proposed assignment + +| field | content | width used | +|---|---|---| +| `classid` (bytes 0..4) | the weather-cell class (§1.4 — **not minted by this plan**) | u32 | +| `HEEL` (4..6) | `(lat_idx >> 6, lon_idx >> 6)` — the **16° × 16° tile**: lat 0..11, lon 0..22 | 2 bytes, one per axis | +| `HIP` (6..8) | `(lat_idx & 63, lon_idx & 63)` — position within the tile | 2 bytes, 6 bits each | +| `TWIG` (8..10) | **0 — dormant, reserved** (zero-fallback: "not consulted", never "compacted") | — | +| bytes 10..16 | **untouched** — V1-legacy u24 tail; new units mint the V3 facet, never `NodeGuid::new`. The bake calls `mint_for(classid_read_mode(c).tail_variant, …)`. | — | + +**The payoff, and it is the headline of this section:** the four hand-picked **16° boxes** the +entire Python arc ran on **are HEEL values**. `substrate_comfort_d_cz_2_7.py`'s `box_index()` +becomes a **key predicate**, and a box read becomes a **HEEL-prefix range scan** — the +"prerender from keys alone, zero value decode" property applied to a physical field. The +65 × 65 = 4225-cell box of the Python is the 64 × 64 HIP space of one HEEL tile (plus the +inclusive edge row/column the probe took). + +`TWIG` stays dormant deliberately: it is the reserved home for a **pressure level** if a later +wave decides a (cell, level) is its own node, or for a sub-0.25° refinement. RESERVE, DON'T +RECLAIM — a non-zero mint later wakes it with **zero `ENVELOPE_LAYOUT_VERSION` change**. + +### §1.3 The three wrinkles — stated, not designed around + +1. **Neither axis tiles evenly at a power-of-two tile size.** 721 = 11 × 64 + 17 (the grid is + pole-inclusive: 720 + 1), and 1440 / 64 = **22.5**. So the last tile on each axis is + **ragged** (17 rows; 32 columns). Decision: **accept ragged tiles**, address by index with + half-open tile ranges, and *never* pad — padding would invent cells that do not exist and + would silently enter any global statistic. `D-WXS-2`'s bar asserts the ragged tiles + round-trip exactly. +2. **Longitude is cyclic; Morton prefix ranges are not.** A box crossing the 0°/360° seam is + **two prefix ranges, not one**. The Python already does `% len(LONS)`; the Rust key API must + return a **range-set**, and the type must make a single-range answer impossible to write by + accident. Latitude has no wrap (it has poles — a different, non-cyclic degeneracy: the pole + *row* is a single physical point repeated 1440 times, which will show as an exact-tie + plateau in any rank statistic; the Python's tie-averaging `spearman` already handles it, + and the Rust side must too). +3. **`MailboxId ≠ NiblePath` in code** (`le-contract.md` §5, discrepancy 3: `MailboxId = u32`, + `NiblePath{path: u64, depth: u8}`, no conversion, no shared trait). The three-tier doc's + *"MailboxId IS the NiblePath"* is doc-only and awaiting a ruling. **This plan must not lean + on it.** A weather cell's key is a `NodeGuid`; whether a HEEL tile is also a mailbox id is + out of scope and is not assumed anywhere below. + +### §1.4 classid — a mint decision, NOT taken here + +`0x0F = Geo` already exists in the OGAR domain table; free domains are `0x03–0x06` +(`weather-substrate-poc-v1` §, carried into v2 §5). The composed classid is canon-high: +`domain ++ appid` in the high u16, **ClassView selector** in the low u16. Minting is an +OGAR-side, operator-gated act — **this plan does not mint it and does not invent a placeholder +that could reach a committed dataset.** `D-WXS-0` (§4, W0) is the blocked row. Until it is +resolved the bake **must refuse to write** rather than write under `0x0000_0000` (the +zero-fallback ladder owns that value and a weather dataset carrying it is unroutable garbage +with no way to tell it apart from a bootstrap row). + +--- + +## §2 The value — which variables ride which facets, in which layout + +### §2.1 Capacity, against the real budget + +Free slab after existing tenants: **292 bytes** (§0.4). As 16-byte facets: **18 facets = +216 payload bytes**. + +| scheme | payload bytes needed | fits as 4+12 facets? | fits as a raw byte lane? | +|---|---|---|---| +| all 122 fields @ 1 B | 122 | **yes** (11 facets = 176 B of slab) | yes | +| 108 time-varying fields @ 1 B (statics split out, §2.4) | 108 | **yes** (9 facets = 144 B) | yes | +| all 122 fields @ 2 B | 244 | **NO** (21 facets = 336 B > 292) | yes (244 ≤ 292) | + +**The headline survives, with the budget corrected: the entire ERA5 state for a cell — all 17 +surface, all 91 upper-air across every one of the 13 pressure levels, all 14 statics — fits +inside one canonical 512-byte `NodeRow` at 1 byte/field, with room to spare. One cell is ONE +node.** No spillover table, no second row, no level-as-node scheme. + +But the 2-byte column matters and must not be skipped: **at 2 bytes/field the state does *not* +fit as proper 4+12 facets** — only as an unprefixed raw byte lane (the shape `FrozenStyle` / +`LearnedStyle` / `ExploreStyle` already use: 12 raw palette bytes, no classid prefix). If +`D-WXS-7` (§5) shows 1 byte/field is insufficient for some variable, the fallback is +**per-variable 2-byte stacked floors on a subset**, not a global 2-byte scheme. + +### §2.2 The layout choice, argued against the §3 catalogue + +Rejected, with reasons: + +| # | layout | why not | +|---|---|---| +| L1/L2/L3 | rails `part_of:is_a` / `memberof:members` / mereology:taxonomy | Relational planes. A cell's temperature is not a relation to anything. The classview is a *focus lens*: relational data focuses rails, and this data is not relational. | +| L5 | `4×(8:8:8)` SPO triplets | The SPO reading is wrong for a physical field. **Kept as the named alternative** for a 3-D wind lane `(u,v,w)` per level if `vertical_velocity` is ever baked alongside u/v — four levels per facet. Not W1. | +| L6 | `3×(8:8:8:8)` SPOG quads | The Odoo factoring. Rejected outright. | +| L7 | `2×48` `hhtl ++ helix` absolute location | This IS location — and for a weather cell **location is the key, not the payload**. Storing it again in the value slab is the duplication §2's "never waste a slot" rule exists to prevent. (The `le-contract` §3 open item — whether the CANON key tail is literally an L7 facet — is untouched here and must not be unified silently.) | +| L8 | `helix ++ CAM_PQ` analog | Retired for this data by measurement, not taste: `weather-normalized-substrate.md` §12.12 **RETIRED the `Pair48` mint**, and §12.12's own regrade fixes the split — **rim = L1-metric carrier; azimuth = render carrier, deliberately not L1-metric**. Routing a weather scalar through the analog lane re-opens the exact "invented round-trip API" the helix crate names twice, unprompted. | +| G1/G2/G3 | wide carvings | **"New classes MUST NOT be born into G1–G3; the waiting room is not a destination."** The weather class is new. | + +**Chosen: L4 — `6 × (8:8)` palette256², one byte per ERA5 field, the pair being a physically +paired couple.** + +Justification in the catalogue's own terms: the classview is the **focus lens the data shape +wants**, and this data is *similarity data over commensurable scalars* — which is precisely +what L4 is for, and precisely what the arc has already measured. The pairing is not +decoration: `(u : v)` in one 8:8 pair means **wind-vector similarity is one 256×256 table +read**; `(msl : sp)`, `(T : q)` at a level, `(T_2m : T_sst)` are the natural couples. Slot +purity is preserved because *which* couple a pair holds is a ClassView fact (§3), never +inspectable from the bytes. + +### §2.3 ⚠ A stated deviation from L4's default reading — and the measurement that licenses it + +`le-contract.md` §3 pins L4's canonical distance/rank read to the **analytic Fisher-z codec** +(`bgz-tensor::fisher_z::{FamilyGamma, FisherZTable}`, certified ρ ≥ 0.999 on cosines). +**For weather scalars that is measured to be WRONG**, on this repo's own committed artifacts: + +- `weather-normalized-substrate.md` §12.1, real ERA5 `2m_temperature`, 1,038,240 gridpoints, + same 256 buckets, same percentile window: **linear MAE 0.0684 K / 0 empty buckets / 115.7 + effective buckets**, vs **Fisher-z-then-linear MAE 0.2168 K / 76 empty buckets / 28.1 + effective**. Fisher-z is **3.2× worse and burns 228 of 256 buckets**. Mechanism measured, not + assumed: `arctanh` moves resolution toward the bounds; ERA5 anomaly has **77 % of mass inside + |s| < 0.25** — mass in the middle. (§12.10 later narrowed this: scored against a noise floor + rather than round-trip, the two are indistinguishable; **what survives is bucket economy, not + a validity failure**. Bucket economy is exactly what a 1-byte-per-field scheme is spending.) +- `le-contract.md` §3 supplies the demarcation itself: *for a **monotone BOUNDED** continuous + field the exit is **analytic, not materialized** —* helix `RollingFloor` + (`quantize` → `bucket_center`), with elevation as the canonical case at **1 byte / RMSE ≈ + 0.11 % of range**. A weather scalar over a robust percentile window is exactly that shape. + +**Decision: the weather ClassView declares an L4 lane whose analytic codebook is the linear +`helix::quantize::RollingFloor`, not the Fisher-z gamma.** Distance is +`helix::distance::DistanceLut::from_floor` — 256 × 256 × u16 = **128 KB, L1/L2-resident**, with +the triangle-inequality regression already shipped (`distance.rs:87-105`, `:118-135`). The +deviation is recorded here in the plan, not smuggled into code, and the `le-contract` §3 note +is the authority for it — not an exception to it. + +**Two hard consequences, both falsifiable:** + +- **Circular variables must NEVER ride an L1 palette byte.** `end_idx` is monotone, so a + wrapped bearing puts 359° and 1° at *maximum* L1 distance (§12.2 point 5 — **stands**, and is + crate-documented as the "raw-azimuth 2π wrap"). Wind therefore rides as **(u, v) components, + two linear bytes**; a bearing, if ever wanted, is derived at read time from the pair. This is + also why `wind_speed` must be baked as its **own byte** and not reconstructed: §6.5 measured + stored speed ≥ `hypot(ū, v̄)` at **100 %** of samples, mean ratio **1.115**, max gap + **14.37 m/s** — the Jensen gap **is gustiness**, real signal. +- **The layout choice must not presuppose `D-WXA-5`'s outcome.** One byte per field is exactly + the quantisation whose fidelity the gate has never tested. Mitigation is structural, not + verbal: **W1 bakes a minimal justified field set, not all 122** (§2.5), the lane is + additive/reserve-don't-reclaim so it grows without a stride change, and `D-WXS-7`'s + graded-resolution ladder (16/64/256 levels) measures the marginal value of a byte *before* + 296 byte-positions are committed. + +### §2.4 The 14 statics — a separate class, a separate dataset, ONE version + +Time-invariant fields (orography family, `geopotential_at_surface`, `land_sea_mask`, +`lake_cover`/`lake_depth`, vegetation ×4, `soil_type`) baked into the versioned row would be +rewritten **92,044 times**: 1,038,240 × 14 B × 92,044 ≈ 1.3 PB of identical bytes. + +**Decision: statics get their own classid and their own dataset, baked once, ONE Lance +version, joined by the identical HEEL/HIP key.** The join is free precisely because the key is +the address. `D-WXS-5`'s bar is a version-count assertion — a bug that rewrites statics per +timestep is caught by `version_count == 1`, not by review. + +### §2.5 The W1 field set — minimal and justified, not all 122 + +Baking 122 fields in W1 would commit 122 byte-positions under an untested quantisation and +would make every downstream bar slower to run for no measured gain. W1 bakes exactly the +fields the open work needs: + +| facet | pair 0 | pair 1 | pair 2 | pair 3 | pair 4 | pair 5 | +|---|---|---|---|---|---|---| +| F0 (surface) | `msl` : `sp` | `u10` : `v10` | `wspd10` : `t2m` | `sst` : `tcwv` | `tcc` : `sic` | reserved-zero | +| F1 (850 hPa) | `u850` : `v850` | `t850` : `q850` | `z850` : `w850` | reserved | reserved | reserved | +| F2 (500 hPa) | `u500` : `v500` | `t500` : `q500` | `z500` : `w500` | reserved | reserved | reserved | + +`msl` + `u10/v10` are what `D-CZ-8` needs (ζ from the 10 m wind). `z500` is the classic +verification target and the field `weather-substrate-poc-v1`'s bake-off named. +Everything else is **reserved-zero**, dormant, expandable per class without a version bump +(`NodeRow` doctrine 1: *clean ⇒ expansion is classid-inherited*). + +### §2.6 Slot purity (§2) — how it is enforced, and where the temptation is + +**Nothing in the payload says what a byte means.** No variable names, no level numbers, no +column ordinals, no display labels. The mapping *"F1 pair 0 hi-byte = `v_component_of_wind` at +850 hPa, unit m/s, floor id 7"* lives in the **ClassView-side field manifest** selected by the +classview u16 — the register-file model: the SoA is dumb bytes, the class makes them +meaningful. + +Three places the temptation is real, and the ruling for each: + +1. **lat/lon.** It is a position, and §2 forbids positions in payload slots. It is **in the + key**, which is the key's job (`HEEL`/`HIP` are the cascade address; the OGAR pin is + explicitly *"the key prerenders nodes with zero value decode"*). Writing lat/lon into the + value slab as well would be both a §2 break and a duplication of the address. Forbidden. +2. **The codebook edges** `(lo, hi, floor_version)` per variable. Per-row they would cost + ~1 KB/row and would be a slot-purity break. They are **dataset (table) metadata**, aligned + to the Lance version boundary — which is exactly `weather-normalized-substrate.md` §4's + recommended operating rule: *calibrate once per epoch, freeze, align any re-roll with a + version boundary, record `(lo, hi, version)` in dataset metadata, so "same code ⇒ same + value" holds exactly within a Lance version and `QueryReference::at` rehydrates with the + floor that produced it.* The floor stamp and the Lance version become the same kind of + object: a calibration epoch. +3. **The timestamp.** It is the **version**, not a payload slot (§3). + +**Honest open item, carried not resolved:** `le-contract.md` §5 discrepancy 6 (via +`weather-normalized-substrate.md` §12.2 point 6, `[G-absence]`) — **there is no per-value-lane +*reading* selector in the contract at all.** `ReadMode` has three axes (tail / value_schema / +edge_codec); `ValueSchema` selects **presence**, not reading. So "the classview selects the L4 +reading" is doctrinally correct and **structurally unimplemented**. This plan does **not** +invent one. `D-WXS-1` ships the manifest as a **committed data artifact** the bake reads, with +its identity stamped into dataset metadata, and flags the missing selector as an +architecture-entropy item for the orchestrator. Anything else would be a contract-surface mint +smuggled in through a weather plan. + +--- + +## §3 The time axis — a timestep is a Lance version + +**Ruling in force:** `E-MARKOV-TEMPORAL-STREAM-1` — *episodic = Lance versions* (the OGAR +D-DELTA mapping, now primary); the ±5 window generalizes to a **version-range read** +(`QueryReference::at(v, rung)` + `deinterlace`) at any width, per-reader rung, replayable, and +**still a projection — zero copies**. + +| need | shipped surface (consumed, never re-implemented) | +|---|---| +| one timestep = one version (writer) | `graph::cycle_sink::LanceCycleWriter` (`open`/`bootstrap`/`head()`/`scan_image`/`reconcile_scans`), `VersionedGraph::commit_encounter_round` | +| time-travel read | `VersionedGraph::at_version(v)` (`versioned.rs:428`) · `current_version()` (`:435`) | +| version-range read | `planner::temporal::{QueryReference::at(v, rung), deinterlace, LanceVersion = u64, EpistemicMode, TemporalStatus, classify}` | +| per-key trajectory | `temporal::{local_trajectories, local_trajectory_of}` | +| diff two timesteps | `GraphDiff` (`versioned.rs:70`), `GraphSealStatus` (`:54`) | + +Two shipped tests already pin the invariant +(`a_whole_cycle_of_casts_is_one_wal_write_one_version`, +`p4a_drains_casts_and_seals_one_wal_write_one_version`). **Any weather deliverable that +re-implements a version writer is the defect, not the feature.** + +**What this replaces, concretely.** `substrate_comfort_d_cz_2_7.py:50` reads +`TIMESTEPS = [54358, 54358 + 4*182, 54358 + 4*365]` — three hand-typed integers with an +`EPOCH`-arithmetic anchor guard. On the substrate that is a **version range**, and the +19-storm set (each at its own `t0`) is a **version set**. The seasonal spacing that made those +three integers a real anti-cherry-pick control becomes a *stated version-selection policy* that +the harness can vary — which is what makes `C1`'s "≥3 independent timesteps" bar cheap to raise +to 30 instead of an argument about fetch budget. + +**Two honest risks, both pre-registered as bars (§4 W2):** + +- **92,044 versions on one dataset is unverified.** CONJECTURE. `D-WXS-6` measures manifest + growth and `at_version(1)` latency at N = 1 / 10 / 100 / 1000 versions and states whether the + growth is sub-linear. **KILL:** superlinear growth ⇒ "one version per timestep" does not + survive 65 years and the time axis needs an explicit chunking policy — a pre-registered + alternative, not a patch discovered mid-run. +- **The forecast/analysis two-lane model** (poc-v2 §1's *"one dataset, versions are cycles"*) + is untouched by this plan and stays as ruled. + +--- + +## §4 Waves, D-ids, and pre-registered bars + +**The rule for every bar below** (`CLAUDE.md` § The falsifiability rule): the bar is written +and **committed before the run**; it has a **control that can lose**; it has a **stay-silent +twin** using a **non-trivial** input (an empty-input silence case proves nothing); and it names +its **KILL** condition. An assertion implied by the code it tests is not a test. + +### W0 — decisions and pre-registration (no executable code) + +| D-id | deliverable | +|---|---| +| **D-WXS-0** | **classid mint** for the weather-cell class + the statics class. **BLOCKED** — OGAR-side, operator-gated. `0x0F = Geo` exists; the appid/classview half is open. **KILL:** if no domain:appid can be assigned, the bake must **refuse to write** — a dataset under `0x0000_0000` is unroutable and indistinguishable from a bootstrap row. Nothing downstream runs against a placeholder. | +| **D-WXS-1** | **the field manifest v1** — the ClassView-side table mapping (facet, pair, byte) → (ERA5 variable, level, unit, floor id). A committed data artifact, not literals in Rust. | +| **D-WXS-1a** | **the variable census, as a committed re-runnable probe** — the `.zmetadata` read behind §0.6. A census asserted in prose is the chat-only-figure defect, third instance in this arc. | + +**Bar B0 (manifest is load-bearing).** *Can-it-fire:* mutating one manifest entry must change +the bytes the bake writes for at least one cell. *Stay-silent twin (non-trivial):* editing a +**reserved-zero** slot's description, or reordering two entries that describe the same +(facet, pair, byte), must change **not one written byte**. **KILL:** if the bake's output is +insensitive to the manifest, the mapping is hard-coded somewhere and slot purity is theatre. + +### W1 — the bake (the missing Zarr → `NodeRow` path) + +| D-id | deliverable | file | +|---|---|---| +| **D-WXS-2** | key codec: `(lat_idx, lon_idx) ↔ NodeGuid` (HEEL/HIP, ragged tiles, lon-wrap range-**set**) | `crates/weather-poc/src/key.rs` | +| **D-WXS-3** | shared canonical floor calibration (per-variable `[lo,hi]` at the 0.4–99.6 percentile over a **global** sample, frozen per epoch, stamped into dataset metadata) | `.../src/floor.rs` | +| **D-WXS-4** | the bake: one timestep slab → 1,038,240 `NodeRow`s → **one Lance version** | `.../src/bake.rs` | +| **D-WXS-5** | statics bake — separate classid, separate dataset, one version | `.../src/statics.rs` | + +**Bar B1 (key).** *Primary:* all 1,038,240 `(lat,lon)` → key → `(lat,lon)` round-trip exactly, +**and no two distinct cells collide on one key**. *Control that can lose:* an off-by-one on +the **ragged last tile** (17 lat rows / 32 lon cols) must make the round-trip fail — if the +suite passes with that injected, it never tested the ragged case. *Stay-silent twin:* a box +entirely inside `[0°, 360°)` must return a **single** range and must **not** report a wrap; +a box crossing the seam must return **exactly two**. **KILL:** any collision ⇒ HEEL/HIP alone +does not address the grid and `TWIG` must be recruited — a design change, reported, not +patched. + +**Bar B2 (floor).** *Primary:* on the global calibration sample, **zero empty buckets** and +saturation below a pre-registered ε per variable (`RollingFloor::occupancy()` **is** the +histogram — no separate instrument). *Control that can lose:* a floor calibrated on **one 16° +box** applied globally must show high saturation — this is `GEO-DEGENERATE` re-homed, and +D-CZ-1 measured that construction saturating **72–97 %** per regime, so it is known-losable. +*Stay-silent twin:* the **global** floor must **not** show high saturation on any single box — +including the storm boxes, where the field is widest. **KILL:** if one global floor cannot +cover a variable within ε, that variable needs its own floor (§4 policy (c) hybrid) and the +"one comparable substrate" claim narrows *for that variable*, stated per-variable rather than +globally. + +**Bar B3 (bake round-trip).** *Primary:* read back version *v*, dequantize via +`bucket_center`, compare against the retained f32 truth — per-variable MAE within the +pre-registered ±½-bucket bound (≈ 0.195 % of span, `quantize.rs:11-18`). Plus the **512 B +verbatim** check (PR #907's compression-detection falsifier reused: the stride clears Lance's +256 B mini-block cutoff, takes the full-zip path, `1.00×`). *Control that can lose:* a bake +whose `floor_version` stamp is omitted must be **detected as unreadable**, never silently +mis-dequantized — this is the `quantize.rs:20-26` contract made into a test. *Stay-silent +twin:* a bake and re-read at the **same** floor version must produce **byte-identical** rows +across two runs (determinism). **KILL:** if the round-trip error exceeds the bucket bound, the +quantisation path has a bug and no fidelity number computed on it means anything. + +**Bar B4 (statics).** *Primary:* after committing N timestep versions to the dynamic dataset, +the statics dataset has **exactly one** version. *Control:* a deliberately re-writing statics +driver must trip it. *Stay-silent:* a legitimate one-off statics re-bake (a new epoch) must +produce exactly **two** and be labelled, not silently tolerated. + +### W2 — the time axis + +| D-id | deliverable | +|---|---| +| **D-WXS-6** | version-range read for a cell's time series: `QueryReference::at(v, rung)` + `deinterlace`; plus the version-scaling measurement (§3) | + +**Bar B5.** *Primary:* a 3-step series read by version range **byte-equals** the same 3 +versions read individually. *Control that can lose:* a `v_ref` outside the committed range must +return **empty**; a rung that excludes a row must **actually** exclude it (both halves — a +filter that never excludes and one that excludes everything carry identical information). +*Stay-silent twin:* a rung that should exclude **nothing** must return the full set — proving +the filter discriminates rather than always firing. **KILL:** as §3. + +### W3 — re-homing `D-WXA-5`, the real gate + +| D-id | deliverable | +|---|---| +| **D-WXS-7** | representation fidelity on the substrate: ρ(code-distance, field-distance) over pairs sampled from the **whole grid**, computed with `jc::reliability::spearman` | +| **D-WXS-8** | cross-variable comparability at grid scale — `p2_probe.py` re-run as a substrate read, ≥4 variables incl. ≥2 distinct units, ≥3 seasons | + +**Bar B6 — `D-WXA-5`, re-specified in three parts.** The poc-v2 bar (ρ ≥ 0.98 + shuffle +control fails) is **kept as the floor and demoted from the verdict**, per §0.3: + +- **(a) primary — ρ ≥ 0.9996**, the bar `p2_probe.py` actually pre-registers and where a real + pair **genuinely failed** (K×K = **0.999556**, below bar; the cross-unit pairs passed at + 0.999736 / 0.999722). A bar with a *demonstrated* failure on real data is a bar that can + fire. +- **(b) control that can lose — the shuffled decode table must score below 0.98.** Known + losable: D-CZ-1 measured `CAL-SHUFFLE` at ρ = **0.003 … 0.159** and RMSE **1498.6 Pa** + against real arms at 0.99999. This is the poc-v2 anti-vacuity control, retained verbatim. +- **(c) can-it-DIFFER — a graded-resolution ladder at 16 / 64 / 256 levels must produce a + strictly monotone ρ ordering, run BEFORE any verdict.** This is the half D-CZ-1 discovered + the hard way (§6.4: real-arm ρ spread 3×10⁻⁶…4.7×10⁻⁵ ⇒ *"C4 could not have fired"*). + **KILL for the metric:** if 16 levels is indistinguishable from 256, ρ is **decorative** on + this data and the gate moves to a **physical-unit** metric (per-variable MAE in K / Pa / + m·s⁻¹, range-normalised) *before* a single fidelity claim is made — exactly the amendment + D-CZ-1 made legitimately in preflight, and which would be illegitimate the moment a verdict + cell had been scored. +- **KILL for the substrate (the thesis-level one, unchanged from poc-v2):** every arm below + ρ ≈ 0.9 ⇒ the hierarchical-prefix-is-ancestry assumption does not survive contact with a + real physical field, and a large part of the substrate's `[H]`/`[S]` map downgrades from + "faithful code" to "useful router." That is a thesis result, not a weather result. + +**Bar B7 (`D-WXS-8`).** *Primary:* cross-unit ρ ≥ 0.9996 on the shared canonical floor. +*Control that can lose:* **per-variable floors must lose on cross-unit pairs** — measured +0.9997 shared vs **0.857–0.875** per-variable, a four-orders-of-magnitude discrimination that +appeared without being tuned for. *Stay-silent twin:* within-variable, shared must **not** +cost resolution — measured ≤ 0.0001 difference, zero empty buckets. **KILL:** if the shared +floor loses to per-variable floors on any cross-unit pair at grid scale, §4 policy (a) is +refuted and the substrate is a per-variable store — which would retract the arc's product +claim, and is the reason this runs early. + +### W4 — re-homing `D-CZ-8` (ζ + a range-normalised transfer metric) + +| D-id | deliverable | +|---|---| +| **D-WXS-9** | ζ = ∂v/∂x − ∂u/∂y as a **substrate read** — a 5-point stencil over neighbour keys; plus the quantisation falsifier below | +| **D-WXS-10** | the **vorticity regime ladder over the whole grid** (regimes = ζ-percentile bands, not four hand-picked boxes) + coverage-matched donor selection + a range-normalised `L` | + +**Why this is a substrate read.** A neighbour in HEEL/HIP address space is `lon_idx ± 1` / +`lat_idx ± 1` — **address arithmetic**, and within a HEEL tile the cells are contiguous. The +stencil that `grad_flat()` currently expresses as `np.gradient` over a Zarr chunk becomes a +key-range read. That is the whole point of §1. + +**Bar B8 — the falsifier this design *needs* and which does not exist anywhere in the arc: +differencing amplifies quantisation error.** ζ is a *difference of differences* of two +1-byte-quantised fields. A ±½-bucket error (≈0.195 % of span) is negligible on a value and can +be **O(1) relative** on a small gradient. *Primary:* ρ(ζ from quantised u,v ; ζ from retained +f32 truth) ≥ a pre-registered bar, reported **per ζ-magnitude decile** (the error will be +worst where ζ is smallest, and a global ρ would hide exactly that). *Control that can lose:* +ζ derived from a **16-level** palette must be visibly worse. *Stay-silent twin:* on a laminar, +near-zero-ζ region the discriminator must **not** flag turbulence — the `closed_class_guess` +150/150 defect in its weather clothing. **KILL:** if quantised ζ does not track true ζ, +**ζ must be baked as its own derived lane** rather than derived at read time — a real design +fork, pre-registered here so it is a decision and not a mid-run rescue. + +**Bar B9 — the range confound, and exactly what the full grid does and does not buy.** + +*What the full grid does NOT buy:* it **does not dissolve §7.9's confound.** `L` was measured +to be essentially a function of **coverage** — `L` vs cell `saturation` at Pearson **+0.917** / +Spearman **+0.833**, and `L̄[T]` vs the regime's own value range at **Spearman +1.000, +perfectly monotone** (R4's implied range ≈ 18× R1's: ~7075 Pa vs ~386 Pa). A foreign fixed +codebook applied to a wider target will saturate more, on 1.04 M cells exactly as on 4225. +Moving to the full grid changes the sample, not the arithmetic. **This plan asserts no +dissolution.** + +*What the full grid DOES buy — three specific, non-rhetorical things:* + +1. **Coverage becomes controllable.** With 1,038,240 cells you can **select** donor and target + populations **matched on value range and offset** by construction. With four hand-picked + boxes you could only observe the confound; with the grid you can hold it constant — which + is the methodological frame (hold variables constant to test the others) applied to the + confound instead of to the regime. +2. **"Donor" can finally mean a *regime* rather than a *range*.** The shared canonical z-floor + (`D-WXS-8`, measured cross-unit ρ 0.9997 vs 0.857–0.875 per-variable) puts every variable on + one commensurable scale, so a donor codebook can be defined by a **ζ-band** rather than by + whatever range four boxes happened to have. +3. **The regime axis stops being n = 4.** ζ-percentile bands over the globe give a regime + sample of millions of cells per band and let `C1`'s "≥3 timesteps" bar become ≥30 without a + fetch-budget argument. It also removes `C5`'s structural blocker's *cause* — though **not** + `C5` itself: the golden index floor needs N ≥ F(17)² = **2,550,409** and the global grid has + **1,038,240** cells, so `GEO-GOLDEN-HI` is **still not constructible**, now short by ~2.5× + rather than by three orders of magnitude. Stated so nobody re-discovers it as a surprise. + +*The bar itself.* **Primary:** the ζ-regime effect on a **range-normalised** `L` must survive +**coverage matching** — report `L` both raw and residualised on `saturation`, and report the +matched-population version alongside. **Control that can lose:** a regime ladder built on a +**shuffled ζ field** (same marginal distribution, destroyed spatial structure) must show **no** +monotone `L̄` trend. **Stay-silent twin:** two donor/target sets drawn from the **same** ζ-band +must show `L̄ ≈ 0` — if band-internal transfer is as costly as cross-band transfer, the bands +are not a regime axis at all. **KILL:** if `L̄` still rank-correlates ρ ≈ 1.0 with range *after* +coverage matching, then the transfer metric **is** a coverage metric, `D-CZ-8`'s premise fails, +and the result is reported as such — **no further normalisation is attempted until it agrees.** +Re-normalising a metric until it stops showing the confound is how a confound becomes a finding. + +### W5 — gated on W3 green: the honest tail + +| D-id | deliverable | +|---|---| +| **D-WXS-11** | **measure** a full-grid bake wall time. The artifact must state the ~8.1 s serial extrapolation as the **prior** and either confirm or correct it in the same document (§0.5). No claim about parallelism — `D-KIA-A2` owns that. | +| **D-WXS-12** | `jc` ↔ `ndarray` reliability agreement — poc-v2 `D-WXB-4` carried over unchanged: identical non-degenerate inputs must agree; the degenerate case must be **reported** (`jc` → `None`), never folded as `0.0`. **`jc` is the authority; `ndarray::hpc` is the SIMD-side mirror.** | + +**Sequencing.** `W0 → W1 → W2 → W3 → (W4, W5)`. **Nothing in W4 or W5 runs before W3's gate +reports.** A negative W3 is the most informative result available and is thesis-relevant rather +than weather-relevant. + +--- + +## §5 The Python probes, classified: substrate read vs numpy function + +Per the brief, every computation in `substrate_comfort_d_cz_0_1.py` and +`substrate_comfort_d_cz_2_7.py` is classified. **"Substrate"** = a shipped primitive exists and +the Python is re-deriving it; **"numpy"** = no substrate equivalent exists and the Python is +correct to be Python; **"probe-side over a substrate read"** = the statistic stays a statistic +but its *input* becomes a key-range scan instead of a Zarr chunk. + +| Python | classification | substrate surface / note | +|---|---|---| +| `encode_decode(values, lo, hi)` — 256 uniform levels, clip-saturate, decode to level centre | **SUBSTRATE** | `helix::quantize::RollingFloor::{uniform, quantize, bucket_center}` — the *identical* linear formula (`quantize.rs:99-108`, `:248-250`). `CAL-ABS` **is** a `RollingFloor` with a donor's bounds. This is the single largest re-derivation in the arc. | +| `cell_metrics` → `occupancy`, `saturation` | **SUBSTRATE** | `RollingFloor::occupancy()` **is** the histogram — *"no separate histogram"* (`quantize.rs:5-9`); saturation = the two rim buckets' share; `drift_score()` is the regime-change telltale, free. | +| `spearman(a,b)` (tie-averaged, `nan` on constant input) | **SUBSTRATE** | `jc::reliability::spearman(&[f64], &[f64]) -> Option` — same tie semantics, and `None` where Python returns `nan`. `jc` is the operator-named authority. | +| `grad_flat` (`np.gradient`, Pa/cell, **no cos(lat)**) | **SUBSTRATE-ADJACENT** — becomes a neighbour-key stencil | ⚠ Carry **both** definitions, labelled with units: the committed ladder used the **flat, no-cos** definition (D-CZ-0 §6.2 identified it from the data at max\|ratio−1\| = 0.069), and it **understates the zonal gradient by 1/cos(lat)** — R3 at 60 N is ~40 % low. Silently switching to the metric-correct definition would break comparability with every committed figure. | +| `box_index(clat, clon)` + `% len(LONS)` | **SUBSTRATE** | HEEL-prefix **range-set** (§1.2/§1.3). The `% 1440` is the seam split. | +| `rank_codec` (256 **quantile** levels in-window) | **NUMPY — no substrate carrier exists** | `RollingFloor` is *linear over a window*, not quantile-spaced. `CAL-RANK` has no shipped Rust equivalent. Either it stays the Python reference arm, or it is a **new codec** owing its own probe. Do not paper over this by calling `RollingFloor` "the rank codec". | +| `fisherz_codec` | **SUBSTRATE** (`helix::fisher_z` / `bgz-tensor::fisher_z`) — but **control-arm only** | §12.1 measured it 3.2× worse and 228/256 buckets burnt on these shapes; §12.10 narrowed that to *bucket economy, not validity*. Keep it as the shape-mismatch control, never as the incumbent. | +| `decay_length_cells`, `gini`, `tail_ratio` (C1c instruments) | **NUMPY** — no substrate equivalent (`jc::stats` has none) | Stay probe-side statistics, but computed **over a substrate read** (a key-range scan), not over a Zarr chunk. | +| `fetch(var, key)` + numcodecs blosc decode | **PYTHON, correctly** | poc-v2 §1's Stage-A **disposable** ingest. Do not gold-plate the throwaway; do not put GRIB2 or a Rust Zarr reader on the critical path. | +| `t_index()` + the `EPOCH` anchor assert | **SUPERSEDED** | The time axis is Lance versioning (§3). The anchor guard's *spirit* survives as a manifest/metadata check that fails loudly if the store is re-chunked. | +| `TIMESTEPS = [...]`, per-storm `t0` | **SUPERSEDED** | A version range and a version set (§3). | + +--- + +## §6 Work split — design/judgment vs mechanical grindwork + +### §6.1 Design / judgment (Opus or the main thread — accumulation, never delegated) + +- The **facet-layout choice** and its stated deviation from L4's default Fisher-z reading + (§2.2/§2.3) — it holds two measured findings and a doctrine note in mind at once. +- The **key assignment** and the three wrinkles (§1) — a wrong tiling is a silent + data-corruption class, not a bug. +- **Every bar in §4** — a bar written by the code's author is the + `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` rule waiting to fire. Bars + are pre-registered by design, committed before the run, and (for any verdict-tier claim) + independently spec-audited. +- The **`D-WXA-5` re-specification** (§0.3 / bar B6) — a judgment call about an instrument's + dynamic range, grounded in a committed measurement. +- The **§7.9 confound treatment** (bar B9) — deciding what the grid buys and what it does not. +- **`D-WXS-0`** (the classid mint) — operator/OGAR-gated, not a worker task. + +### §6.2 Mechanical grindwork (Sonnet workers, one file each, disjoint) + +Every worker brief pastes `.claude/v3/knowledge/sonnet-worker-guardrails.md` §1 **verbatim**, +plus §2 (vocabulary disambiguation) and §5 (STOP+report triggers). No grindwork spawn without +it. Every brief also carries: *read `.claude/board/AGENT_LOG.md` before starting; do NOT write +it — leave your record in your own tag-file under `probes/weather-p1/exec-runs/`; the +orchestrator consolidates.* + +| worker | file (sole owner) | task shape | +|---|---|---| +| W-key | `crates/weather-poc/src/key.rs` | encode/decode + range-set; bar B1's tests | +| W-floor | `crates/weather-poc/src/floor.rs` | drive `RollingFloor` calibration from a sample; emit `(lo,hi,version)`; bar B2 | +| W-lane | `crates/weather-poc/src/lane.rs` | L4 pack/unpack **given** the manifest; zero domain knowledge | +| W-bake | `crates/weather-poc/src/bake.rs` | slab → `NodeRow`s → `LanceCycleWriter`; bar B3 | +| W-statics | `crates/weather-poc/src/statics.rs` | the one-version statics path; bar B4 | +| W-stencil | `crates/weather-poc/src/stencil.rs` | neighbour-key arithmetic + ζ; bar B8 | +| W-metrics | `crates/weather-poc/src/metrics.rs` | `jc` battery wiring; bars B6/B7 | +| W-tests | `crates/weather-poc/tests/.rs` | one file per bar, never shared | + +**Orchestrator-only files (shared — a worker touching one is the defect):** +`crates/weather-poc/src/lib.rs` (the `mod` lines), `crates/weather-poc/Cargo.toml`, the root +`Cargo.toml` (**not touched at all** — `weather-poc` is workspace-**EXCLUDED**, on the same +pattern as `jc`, `helix`, `sigker`, `perturbation-sim`), +`crates/lance-graph-contract/src/canonical_node.rs` (the `ValueTenant` enum + `VALUE_TENANTS` — +a canon act, Opus/orchestrator only, and note discriminant **15 is reserved for +`BoardAggregates`**; a weather tenant takes **16**, with its offset **derived** via +`value_offset()` and never written as a literal), and every `.claude/board/*` file. + +### §6.3 Crate scaffold constraints (non-negotiable) + +- **No new repositories.** `crates/weather-poc`, workspace-**EXCLUDED**, on the + `perturbation-sim` template. Verified via + `cargo test --manifest-path crates/weather-poc/Cargo.toml`. +- **Zero-dep default build.** Any `ndarray` dependency is **optional, behind an off-by-default + feature, and sourced by GIT URL — never `path`** (an optional *path* dep is read at manifest + resolution and breaks a clean checkout even with the feature off: codex P2 on #504, the + helix-#460 lesson, recorded verbatim in `perturbation-sim/Cargo.toml`). +- **SIMD only via `ndarray::simd::*`** (`ndarray/src/lib.rs:241`, `src/simd.rs`). **Never + `ndarray::hpc::*` from a consumer**, and never raw intrinsics. +- **The lance family is upstream-authoritative and moves in exact lockstep.** No version bump + is proposed by this plan: `lance = "=9.0.0"` family, `lancedb = "=0.33.0"`, `arrow 58`, + `datafusion 54` **and** `53` (both required — `deltalake-core 0.32.4` pins 53 upstream; + collapsing the lock to one breaks the `delta` feature), rust `1.97.1`. +- **Credentials by name only** — `dev_s3_env::s3_options()` reads the set and returns `None` + when any is missing; on a remote path that must be a **hard error**, never a silent local + fallback. **S3 hydrates; a local mmap-capable directory stores** (#901); a network mount that + looks local is the named trap. + +--- + +## §7 Honest grading + +| claim | grade | provenance | +|---|---|---| +| 1,038,240 cells × 512 B = 0.495 GiB/timestep | `[G]` | arithmetic on the const-asserted stride | +| 122 fields fit one 512 B row at 1 B/field | `[G]` | arithmetic against the **292 B** free budget (§0.4), not 384 | +| 122 fields do **not** fit as 4+12 facets at 2 B/field | `[G]` | 21 × 16 = 336 > 292 | +| HEEL/HIP addresses 721×1440 with 5–6 of 12 quaternary levels/axis | `[G]` | 4⁵ ≥ 721, 4⁶ ≥ 1440 | +| the axes tile raggedly (721 = 11×64+17; 1440/64 = 22.5) | `[G]` | arithmetic | +| shared canonical z-floor beats per-variable floors cross-unit (0.9997 vs 0.857–0.875) | `[H]` | `p2_probe.py`, one timestep, 3 variables, §12.8 — graded `[H]` there pending seasons + a 4th unit | +| the K×K pair (0.999556) is **below** the 0.9996 bar | `[G]` | measured, corrected under review on #920 | +| Fisher-z burns 228/256 buckets on ERA5 temperature | `[G]` on the buckets; `[H]` on the verdict | §12.1 measured; §12.10 narrowed to bucket economy, not validity | +| ρ saturates on the diagonal (real-arm spread 3e-6…4.7e-5) | `[G]` | D-CZ-1, `substrate_comfort_d_cz_0_1.json` | +| `L` vs `saturation` Pearson +0.917; `L̄` vs range Spearman +1.000 | `[G]` | §7.9, committed JSON | +| **`D-WXA-5`'s ρ ≥ 0.98 bar is at risk of being vacuous** | **`[H]` — inference from the two rows above; the reason W3 part (c) exists** | §0.3 | +| ~8.1 s per full-grid timestep | **CONJECTURE** — serial extrapolation from ONE measured cycle (514 ms, 65,536 owners); `D-KIA-A2` Queued/unbuilt; and a bake is not a mailbox cycle | §0.5 | +| 92,044 Lance versions on one dataset is viable | **CONJECTURE** — unmeasured; `D-WXS-6` is the falsifier | §3 | +| L4 + linear `RollingFloor` is the right lane for weather scalars | `[H]` — composed from measured pieces, unrun **as a lane** | §2.2/§2.3 | +| ζ derived from 1-byte-quantised u,v is usable | **CONJECTURE — and the plan's most likely failure point** | bar B8 | +| the grid dissolves the §7.9 confound | **FALSE, asserted nowhere** — it buys *control*, not dissolution | bar B9 | +| `GEO-GOLDEN-HI` becomes constructible at grid scale | **FALSE** — needs 2,550,409; grid has 1,038,240 | §4 W4 | + +--- + +## §8 What contradicts the commissioning framing (consolidated) + +1. **`D-WXA-5` is not on `STATUS_BOARD.md`; no `D-WXA-*` row is** (§0.1). It lives only in the + plan file and the `INTEGRATION_PLANS` prose. +2. **The "C2 gate" label belongs to a different plan** (§0.2). +3. **`D-WXA-5` as written is probably unable to fail** on this data (§0.3) — the arc's own + committed measurement says so. +4. **"32 facets / 33.2 M facets per timestep" overstates the usable budget by ~78 %** — the + real free budget is 292 bytes / 18 facets (§0.4). +5. **The ~8.1 s figure is both an unverified extrapolation and a category error** — a bake is a + columnar write, not 16 mailbox seal cycles (§0.5). +6. **`crates/weather-poc` does not exist** — confirmed against the crate listing; and + `weather-substrate-poc-v2.md` indeed specifies **no bake step**, which is the direct cause + of the arc staying in Python. +7. **`helix`/`Signed360` is not a wind carrier and the `Pair48` mint is RETIRED** (§12.12) — + any design that reaches for `HelixResidue` for weather is re-opening a four-corrections-deep + dead end. This plan does not. +8. **There is no per-value-lane *reading* selector in the contract** (`[G-absence]`, §2.6) — so + "the classview selects the L4 reading" is doctrine without a mechanism. Flagged, not + invented around. + +--- + +## §9 `weather-substrate-poc-v2.md`, part by part + +| part | verdict | +|---|---| +| §1 disposable Stage-A vs permanent Stage-C ingest split | **SURVIVES** — adopted verbatim | +| §1 "versioning is CONSUMED, not built" | **SURVIVES** — §3 consumes exactly that surface | +| §1 one dataset, versions are cycles | **SURVIVES** | +| §1 S3 hydrates / local mmap stores (#901) | **SURVIVES** | +| §2 `D-WXA-1` Stage-A ingest | **SURVIVES**, re-scoped: emit the **shared `soa:*` metadata block** and the field manifest, not just a flat slab | +| §2 `D-WXA-2` `crates/weather-poc` scaffold, workspace-EXCLUDED | **SURVIVES — and is `D-WXS-4`'s prerequisite. NEVER BUILT.** | +| §2 `D-WXA-3` encoder arms A–E | **SUPERSEDED in part.** Arms A/B (helix48, helix+residue) are retired for scalars by §12.1/§12.12. Arm E (`cascade_key`) is **not an encoder arm at all** — it is the **key design** (§1). What remains a live bake-off is **arm D (bgz17 hierarchical palette) vs the measured incumbent (linear `RollingFloor` + `DistanceLut`)**, which is also `F-1`. | +| §2 `D-WXA-4` the `jc` battery | **SURVIVES** — `D-WXS-7`/`D-WXS-8` | +| §2 `D-WXA-5` the gate | **SURVIVES as the floor, re-specified as the verdict** (§0.3, bar B6). **NEVER RUN.** | +| §3 `D-WXB-1..4` ndarray parity / no-silent-scalar / throughput / jc↔ndarray divergence | **SURVIVES**, deferred to W5 (`D-WXS-11`, `D-WXS-12`). **NEVER BUILT.** | +| §4 `D-WXC-1..5` permanent ingest, two-stage retrieval, external score, three lanes | **SURVIVES as future work**, unchanged and still gated behind the representation gate. **NEVER BUILT.** | +| §5 repos / zero new repositories | **SURVIVES** | +| §6 pins (incl. the ⊘ double correction on datafusion 54 **and** 53) | **SURVIVES** | +| §7 credentials | **SURVIVES** | +| §0 C1/C2/⊘C3 (GRIB2 gone; `ecmwf-opendata` is Phase C; the invented Zarr object name) | **SURVIVES**, and ⊘C3's apparatus lesson — *a title match is not an existence check; list the bucket* — is the reason `D-WXS-1a` commits the census as a probe | + +**Net:** poc-v2's *structure* survives almost entirely. What it was missing is the thing this +plan adds — **the bake** — and what it got wrong is one bar's threshold, which the arc's own +later measurement exposed. + +--- + +## §10 Cross-references + +`.claude/v3/soa_layout/le-contract.md` §2/§3/§3a/§5 · `CLAUDE.md` § CANON — Minimal SoA node, +§ The falsifiability rule, § Mandatory Board-Hygiene Rule, § Key Dependencies · +`/home/user/OGAR/CLAUDE.md` § Tier interpretation — 256×256 CENTROID TILE, § the 3×4 path · +`.claude/plans/weather-substrate-poc-v2.md` · `.claude/plans/substrate-comfort-zones-v1.md` +§7.8/§7.9 · `.claude/knowledge/weather-normalized-substrate.md` §3/§4/§5/§8/§9/§12.1/§12.2/ +§12.6/§12.8/§12.10/§12.12 · `.claude/knowledge/encoding-ecosystem.md` (mandatory, read) · +`.claude/knowledge/helix-cartesian-vs-fisher2z.md` (the doctrine §12.12 was retired for not +reading) · `.claude/knowledge/s3-hydration-lifecycle.md` · +`.claude/v3/knowledge/sonnet-worker-guardrails.md` · `.claude/board/EPIPHANIES.md` +`E-MARKOV-TEMPORAL-STREAM-1`, `E-V3-FACET-4-PLUS-12`, `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`, +`E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1`, +`E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` · +`.claude/board/AGENT_LOG.md` 2026-08-05 (PROBE-IGNITION-64K) · +iron rules `I-VSA-IDENTITIES`, `I-NOISE-FLOOR-JIRAK`, `I-LEGACY-API-FEATURE-GATED`. + +--- + +## §11 Board rows landed with this plan + +> Written by the orchestrator (sole writer of `.claude/board/*`) in the same commit as this +> plan, per the Mandatory Board-Hygiene Rule. §11.1 is the `STATUS_BOARD.md` block; §11.2 is +> the `INTEGRATION_PLANS.md` entry. + +### §11.1 `STATUS_BOARD.md` — the D-WXS block + +Landed as its own section. Every row `Queued` except `D-WXS-0` (**Blocked**, operator/OGAR). +Carries two standing notes: the poc-v2 ladder's board-hygiene gap (`grep -c WXA` = 0, nothing +ever built), and that the global grid does **not** unblock `GEO-GOLDEN-HI` (needs 2,550,409; +grid has 1,038,240). + +### §11.2 `INTEGRATION_PLANS.md` — PREPEND entry + +Landed with the plan, recording: the bake as the closed gap, the HEEL/HIP key design and its +three wrinkles, the L4 + linear-`RollingFloor` lane with its stated deviation, the corrected +292-byte capacity budget, statics-as-own-class, timestep-as-version, the `D-WXA-5` +re-specification, and the §7.9 position (the grid buys control, not dissolution). From 010da851fbc09b418c2fe47c341c95ecd95c020f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 09:25:19 +0000 Subject: [PATCH 6/8] weather-poc: the crate scaffold (orchestrator-only files) The crate `weather-substrate-poc-v2.md` named but never created. Workspace- EXCLUDED, on the jc / sigker / helix / perturbation-sim pattern; verified via `cargo test --manifest-path crates/weather-poc/Cargo.toml`. GENUINELY ZERO-DEP, and that is a decision rather than an accident. helix's `quantize::RollingFloor` is the shipped carrier for the linear 256-level floor this crate needs, but helix carries a MANDATORY ndarray git dep, and an *optional path* dep is read at manifest resolution -- so even offering `helix = { path = "../helix", optional = true }` would break a clean checkout with the feature off. That is the codex-P2 trap on #504 / the helix-#460 lesson. So floor.rs will carry the linear formula itself, documented as identical to helix's with line references, and the AGREEMENT between the two becomes a measured cross-manifest comparison -- the same shape D-WXS-12 uses for jc <-> ndarray, where parity is a deliverable and not a dependency edge. The reasoning is in the crate manifest so the next session does not re-derive it, along with the recipe if ndarray::simd is ever needed (optional, off-by-default, GIT-sourced -- never a path). Only orchestrator-owned files here: the manifest, lib.rs's mod lines, the root exclude entry, and three one-line placeholders that the worker fleet owns (key.rs / floor.rs / manifest.rs, one sole owner each per plan section 6.2). Cargo.lock is committed, matching jc / onebrc-probe / perturbation-sim. fmt clean; `cargo test` green on the empty crate (0 tests -- an empty pass, and said as such). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- Cargo.toml | 7 +++++ crates/weather-poc/Cargo.lock | 7 +++++ crates/weather-poc/Cargo.toml | 40 +++++++++++++++++++++++++++++ crates/weather-poc/src/floor.rs | 1 + crates/weather-poc/src/key.rs | 1 + crates/weather-poc/src/lib.rs | 41 ++++++++++++++++++++++++++++++ crates/weather-poc/src/manifest.rs | 1 + 7 files changed, 98 insertions(+) create mode 100644 crates/weather-poc/Cargo.lock create mode 100644 crates/weather-poc/Cargo.toml create mode 100644 crates/weather-poc/src/floor.rs create mode 100644 crates/weather-poc/src/key.rs create mode 100644 crates/weather-poc/src/lib.rs create mode 100644 crates/weather-poc/src/manifest.rs diff --git a/Cargo.toml b/Cargo.toml index 0c3846a9b..2ea11c1c5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -109,6 +109,13 @@ exclude = [ # crates/onebrc-probe/README.md. Verify via # `cargo test --manifest-path crates/onebrc-probe/Cargo.toml`. "crates/onebrc-probe", + # ERA5 0.25-degree global grid baked onto the canonical 512-byte NodeRow -- + # one cell = one node, one timestep = one Lance version. Standalone and + # genuinely zero-dep (see the crate manifest for why helix is NOT an + # optional path dep). Verify via + # `cargo test --manifest-path crates/weather-poc/Cargo.toml`. + # Plan: .claude/plans/weather-soa-bake-v1.md. + "crates/weather-poc", ] resolver = "2" diff --git a/crates/weather-poc/Cargo.lock b/crates/weather-poc/Cargo.lock new file mode 100644 index 000000000..fdaedaca0 --- /dev/null +++ b/crates/weather-poc/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "weather-poc" +version = "0.1.0" diff --git a/crates/weather-poc/Cargo.toml b/crates/weather-poc/Cargo.toml new file mode 100644 index 000000000..681d9acf2 --- /dev/null +++ b/crates/weather-poc/Cargo.toml @@ -0,0 +1,40 @@ +[package] +name = "weather-poc" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" +publish = false +description = "Zarr -> NodeRow: the ERA5 0.25-degree global grid baked onto the canonical 512-byte SoA row. One cell = one node (HEEL = 16-degree tile, HIP = within-tile), one timestep = one Lance version. Plan: .claude/plans/weather-soa-bake-v1.md" + +# Standalone research crate, EXCLUDED from the lance-graph workspace (same +# pattern as jc / sigker / helix / perturbation-sim). Verify with: +# cargo test --manifest-path crates/weather-poc/Cargo.toml +# +# ── ZERO-DEP BY CONSTRUCTION (an orchestrator decision, plan §6.1) ─────────── +# This crate has NO dependencies, not even optional ones, and that is +# deliberate rather than incidental: +# +# * `helix::quantize::RollingFloor` is the shipped carrier for the linear +# 256-level floor this crate needs (plan §5 classifies the arc's Python +# `encode_decode` as a re-derivation of exactly it). But `helix` carries a +# MANDATORY ndarray git dep, and an *optional path* dep is read at manifest +# resolution -- so merely offering `helix = { path = "../helix", optional }` +# would break a clean checkout even with the feature off. That is the +# codex-P2 trap on #504 / the helix-#460 lesson, named in plan §6.3. +# * So `floor.rs` carries the linear formula itself, documented as identical +# to helix's, and the AGREEMENT between the two is a separate deliverable +# run across manifests -- the same shape as D-WXS-12 (jc <-> ndarray), where +# parity is a measured comparison rather than a dependency edge. +# +# If a future wave needs `ndarray::simd`, add it the way perturbation-sim does: +# optional, behind an off-by-default feature, sourced by GIT URL -- never a path. +# All SIMD comes from `ndarray::simd` per the workspace rule, never raw +# intrinsics, and never `ndarray::hpc::*` from a consumer. + +[dependencies] + +[dev-dependencies] + +[lib] +name = "weather_poc" +path = "src/lib.rs" diff --git a/crates/weather-poc/src/floor.rs b/crates/weather-poc/src/floor.rs new file mode 100644 index 000000000..96c93dcae --- /dev/null +++ b/crates/weather-poc/src/floor.rs @@ -0,0 +1 @@ +//! Placeholder — owned by the floor worker (plan §6.2). Filled in W1. diff --git a/crates/weather-poc/src/key.rs b/crates/weather-poc/src/key.rs new file mode 100644 index 000000000..be9547984 --- /dev/null +++ b/crates/weather-poc/src/key.rs @@ -0,0 +1 @@ +//! Placeholder — owned by the key worker (plan §6.2). Filled in W1. diff --git a/crates/weather-poc/src/lib.rs b/crates/weather-poc/src/lib.rs new file mode 100644 index 000000000..40d5801e8 --- /dev/null +++ b/crates/weather-poc/src/lib.rs @@ -0,0 +1,41 @@ +//! Zarr → `NodeRow`: the ERA5 0.25° global grid on the canonical SoA row. +//! +//! **Plan:** `.claude/plans/weather-soa-bake-v1.md`. This crate exists because +//! the weather R&D arc ran entirely as Python over a Zarr file — one variable, +//! four hand-picked 16° boxes, three hand-typed timesteps — and the reason was +//! structural: the Zarr→`NodeRow` path did not exist and no plan specified one. +//! +//! # The shape, in four facts +//! +//! * **One cell is one node.** The ERA5 0.25° grid is 721 × 1440 = +//! **1,038,240 cells**; the canonical row is 512 bytes; so a full global +//! timestep is **0.495 GiB**. All 122 ERA5 fields per cell (17 surface + 7 +//! pressure-level × 13 levels + 14 static) fit at 1 byte/field inside the +//! **292 free slab bytes** — see [`crate::key`] and the plan §2.1 for why the +//! budget is 292 and not 384. +//! * **The key is the address.** `HEEL` = the 16° × 16° tile, `HIP` = position +//! within it, `TWIG` dormant-reserved. A lat/lon grid is the literal-x/y case +//! of the 3×4 cascade, so the arc's hand-picked 16° boxes become **HEEL-prefix +//! range scans** rather than array slicing. +//! * **One timestep is one Lance version.** Per `E-MARKOV-TEMPORAL-STREAM-1`, +//! episodic = Lance versions; a time series is a version-range read. This +//! crate never writes a version writer of its own. +//! * **Nothing in the payload says what a byte means.** The (facet, pair, byte) +//! → (variable, level, unit, floor) mapping is a ClassView-side manifest +//! ([`crate::manifest`]), never a slot in the row — the le-contract §2 +//! slot-purity rule. +//! +//! # Status +//! +//! Wave W1 is under construction. `D-WXS-0` (the classid mint) is **blocked** +//! on an OGAR-side, operator-gated decision, and until it resolves the bake +//! must **refuse to write** rather than emit rows under `0x0000_0000` — that +//! value belongs to the zero-fallback ladder and a dataset carrying it is +//! indistinguishable from a bootstrap row. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +pub mod floor; +pub mod key; +pub mod manifest; diff --git a/crates/weather-poc/src/manifest.rs b/crates/weather-poc/src/manifest.rs new file mode 100644 index 000000000..1ed6fc19f --- /dev/null +++ b/crates/weather-poc/src/manifest.rs @@ -0,0 +1 @@ +//! Placeholder — owned by the manifest worker (plan §6.2). Filled in W1. From a8a501bc729fb9a1d5afbbf51b5e62627485ca45 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 09:29:34 +0000 Subject: [PATCH 7/8] D-WXS-1a + D-WXS-2 SHIPPED: the census terminates at an artifact, the key codec is disable-verified Two W0/W1 deliverables land, both gated centrally by the orchestrator rather than taken from the workers' reports. D-WXS-1a -- the ERA5 variable census as a re-runnable probe. Three times in this arc a figure was asserted in prose and later found stale or mis-attributed; the rule that came out of it is that an audit must terminate at an artifact. The census was the last set of numbers in the plan with nothing behind it. Now: `era5_variable_census.py` reads the store's .zmetadata and emits the JSON, and its `--selftest` asserts all 10 constants the plan carries (17 surface, 7x13=91 pressure-level, 14 static, 122 fields/cell, 1,038,240 cells, 92,044 timesteps, 126,665,280 values/timestep, 531,578,880 bytes/timestep). Re-run independently here: PASS. Guard disable-verified -- breaking one expected constant exits 1 with the correct "the fetched answer is authoritative" message, so the selftest discriminates rather than trivially passing. 0 arrays failed to classify. D-WXS-2 -- the key codec. (lat,lon) <-> the canonical 16-byte layout, HEEL = the 16-degree tile, HIP = within-tile, TWIG dormant, tail untouched; ragged tiles addressed by index and never padded; a seam-crossing box returns a range SET, never one range. 5/5 green including an exhaustive 1,038,240-cell round-trip that is also collision-checked. Bar B1's halves disable-verified by me, not asserted by the worker: zeroing the HIP lat byte kills 3 tests (collision, ragged, out-of-range), and removing the seam split kills the wrap twin while the non-wrap twin stays green -- so the pair discriminates in both directions. An UNDECLARED DEVIATION surfaced during implementation and is recorded rather than quietly fixed (plan section 1.3a, board D-WXS-2a). Section 1.2 assigns one WHOLE byte per axis; OGAR's cascade doctrine -- the same passage section 1.1 leans on -- specifies the axis bytes nibble-interleaved (Morton). The plan deviated from the canon it cites and did not say so. What survives: the load-bearing claim, that a 16-degree box becomes a HEEL-prefix range scan, holds under BOTH layouts, because fixing both HEEL bytes fixes a prefix either way. The is_ancestor_of = containment argument also survives -- interleaving governs how two axes share a tier's bits, not per-tier ancestry. What differs: byte-lexicographic order over whole-byte axes is row-major, so a lon+/-1 neighbour is adjacent in key order while a lat+/-1 neighbour is 1440 cells away. Under Morton both are near, which matters for the zeta stencil. Not changed now, deliberately: the prefix claim holds as written, which layout is better for this workload is measurable rather than obvious, and rewriting a spec the moment a worker surfaces a consequence is how a correction starts building. D-WXS-2a pre-registers the comparison; until it runs, section 1.2's layout is a stated deviation and not a ruling, and anything downstream assuming Morton locality is unfounded. Also flagged, resolved by convention and not by test: box_ranges treats lon_lo == lon_hi as wrap-the-whole-circle rather than empty. Neither reading is forced by the spec; it needs a pin or an API that cannot express the ambiguity. Gates: `cargo test --manifest-path crates/weather-poc/Cargo.toml` 5/5; clippy -D warnings clean; fmt clean on the committed files. The floor and manifest deliverables are still in flight and are NOT in this commit. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/STATUS_BOARD.md | 5 +- .claude/plans/weather-soa-bake-v1.md | 44 +++ crates/weather-poc/src/key.rs | 349 +++++++++++++++++++- probes/weather-p1/era5_variable_census.json | 75 +++++ probes/weather-p1/era5_variable_census.py | 291 ++++++++++++++++ 5 files changed, 761 insertions(+), 3 deletions(-) create mode 100644 probes/weather-p1/era5_variable_census.json create mode 100644 probes/weather-p1/era5_variable_census.py diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index ca527a363..3eaef02b1 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -10,8 +10,9 @@ with a control that can lose and a stay-silent twin. |---|---|---|---|---| | D-WXS-0 | classid mint for the weather-cell + statics classes (OGAR-side; `0x0F = Geo` exists, appid/classview open) | 0 | **Blocked (operator/OGAR)** | the bake must REFUSE to write under `0x0000_0000` | | D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | Queued | slot purity §2; bar B0 | -| D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | Queued | ends the chat-only-figure defect for the census | -| D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | Queued | a 16° box becomes a HEEL-prefix scan; bar B1 | +| D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | +| D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | +| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | | D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | Queued | bar B2 | | D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | | D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | diff --git a/.claude/plans/weather-soa-bake-v1.md b/.claude/plans/weather-soa-bake-v1.md index cb2d5dd29..89b2f24ff 100644 --- a/.claude/plans/weather-soa-bake-v1.md +++ b/.claude/plans/weather-soa-bake-v1.md @@ -154,6 +154,50 @@ RECLAIM — a non-zero mint later wakes it with **zero `ENVELOPE_LAYOUT_VERSION` on it.** A weather cell's key is a `NodeGuid`; whether a HEEL tile is also a mailbox id is out of scope and is not assumed anywhere below. +### §1.3a ⚠ UNDECLARED DEVIATION from the OGAR cascade doctrine — whole-byte axes, not nibble-interleaved (found 2026-08-13 by the D-WXS-2 worker, during implementation) + +§1.2 above assigns **one whole byte per axis** (`HEEL` = byte 4 lat, byte 5 lon; +`HIP` = byte 6 lat, byte 7 lon). OGAR's cascade doctrine — the same passage §1.1 +leans on for the 256×256 tile reading — specifies the two axis bytes +**nibble-interleaved**, and names the consequence explicitly: *"the x/y +nibble-interleave = alternating-axis refinement (Morton in centroid space)."* +§1.2 therefore **deviates from the canon it cites, and did not say so.** Recorded +here rather than silently corrected. + +**What survives unchanged.** The plan's actual load-bearing claim — +*a 16° box becomes a HEEL-prefix range scan* — holds under **both** layouts: +fixing both HEEL bytes fixes a prefix either way, so a tile-aligned box is one +contiguous byte range regardless. The `is_ancestor_of` = containment argument in +§1.1 also survives: interleaving governs how two axes **share** a tier's 16 bits, +not the per-tier ancestry the identity mipmap supplies. + +**What actually differs.** Byte-lexicographic order over whole-byte axes is +**row-major** (lat-major, lon-minor), not locality-preserving. Consequence for +`D-WXS-9` (the ζ stencil): a `lon ± 1` neighbour is adjacent in key order, while +a `lat ± 1` neighbour is 1440 cells away. Under Morton both are near. An +arbitrary (non-tile-aligned) box is also not one contiguous range under either +layout, but Morton bounds it in far fewer ranges. + +**Not changed now, deliberately.** Three reasons: the prefix-scan claim holds as +written; which layout is better *for this workload* is a measurable question and +this arc's discipline is measure-don't-assume; and rewriting a spec the moment a +worker surfaces a consequence is how a correction starts building +(`E-ACTOR-IS-NOT-THE-PHASE-PATH-1`'s recorded lesson: *when correcting an +over-built design, the correction must not itself build*). The shipped +`key.rs` implements §1.2 as specified and its five falsifiers are +disable-verified green. + +**The decision this owes.** A pre-registered comparison — row-major vs Morton +over the ζ-stencil neighbour read (`D-WXS-9`) and over a non-tile-aligned box +scan — with the metric stated before the run. Until it runs, §1.2's layout is +**a stated deviation, not a ruling**, and anything downstream that assumes Morton +locality is unfounded. Tracked as `D-WXS-2a` on `STATUS_BOARD.md`. + +**A second, smaller item from the same worker, resolved by convention and not by +test:** `box_ranges` treats `lon_lo == lon_hi` as *wrap the whole circle*, not as +*empty box*. Neither reading is forced by the spec. It needs a test pinning the +chosen one, or an API that makes the ambiguity unrepresentable. + ### §1.4 classid — a mint decision, NOT taken here `0x0F = Geo` already exists in the OGAR domain table; free domains are `0x03–0x06` diff --git a/crates/weather-poc/src/key.rs b/crates/weather-poc/src/key.rs index be9547984..453cde51a 100644 --- a/crates/weather-poc/src/key.rs +++ b/crates/weather-poc/src/key.rs @@ -1 +1,348 @@ -//! Placeholder — owned by the key worker (plan §6.2). Filled in W1. +//! Grid cell (lat_idx, lon_idx) <-> the canonical 16-byte `NodeGuid` key. +//! +//! Owner: the key worker (plan `.claude/plans/weather-soa-bake-v1.md` §6.2, +//! deliverable D-WXS-2). Grid: ERA5 0.25 degree, pole-inclusive latitude +//! (`lat_idx` 0..721), cyclic longitude (`lon_idx` 0..1440). +//! +//! # Byte layout +//! +//! This crate is deliberately zero-dep (`Cargo.toml`'s "ZERO-DEP BY +//! CONSTRUCTION" note) and therefore has no dependency on +//! `lance-graph-contract`'s `NodeGuid` type. [`encode_key`] instead emits a +//! raw `[u8; 16]` in the **same byte layout** `NodeGuid` uses (`CLAUDE.md` +//! § "CANON -- Minimal SoA node"), little-endian throughout: +//! +//! | bytes | content | +//! |---|---| +//! | `0..4` | `classid: u32`, little-endian -- a **parameter this module accepts**; it is never composed, minted, or bit-manipulated here (`D-WXS-0`, the classid mint, is blocked on an OGAR-side decision -- plan §1.4) | +//! | `4` | HEEL, lat axis: `(lat_idx >> 6) as u8` | +//! | `5` | HEEL, lon axis: `(lon_idx >> 6) as u8` | +//! | `6` | HIP, lat axis: `(lat_idx & 63) as u8` | +//! | `7` | HIP, lon axis: `(lon_idx & 63) as u8` | +//! | `8..10` | TWIG -- `0u16`, dormant/reserved (zero-fallback: "not consulted", never "compacted") | +//! | `10..16`| untouched (all zero) -- the V1-legacy tail; this crate mints nothing there | +//! +//! **Agreement between this layout and the real `NodeGuid` (`canonical_node.rs`) +//! is NOT verified by this module.** That is a separate, cross-manifest parity +//! deliverable of the same shape as `D-WXS-12` (jc <-> ndarray) -- a measured +//! comparison run across both crates, not a dependency edge from this +//! (deliberately dependency-free) crate. +//! +//! # The three wrinkles this module implements (plan §1.3) +//! +//! 1. **Ragged tiles, never padded.** `721 = 11*64 + 17` and +//! `1440 / 64 = 22.5`: the last tile on each axis is partial (17 lat rows, +//! 32 lon columns). Cells are addressed by index with half-open ranges; +//! nothing is padded to a full tile, because a padded cell would be a +//! cell that does not exist. +//! 2. **Longitude is cyclic, latitude is not.** A box that crosses the +//! 0deg/360deg seam is decomposed into two ranges by [`box_ranges`], never +//! approximated as one. Latitude has no such wrap -- it has poles instead +//! (a pole row is one physical point repeated `LON_COUNT` times, which is +//! a downstream statistics concern, not a key-encoding one). +//! 3. **No `MailboxId`/`NiblePath` here.** That mapping is unresolved +//! elsewhere in the workspace (`le-contract.md` §5) and this module does +//! not lean on it. + +/// Number of latitude rows in the ERA5 0.25 degree, pole-inclusive grid +/// (`720 + 1`). `lat_idx` ranges over `0..LAT_COUNT`. +/// +/// `721 = 11 * 64 + 17`: the grid does not tile evenly at the 64-row tile +/// size, so the last latitude tile (HEEL = 11) is ragged, holding only the +/// 17 rows `704..721` (plan §1.3 wrinkle 1). +pub const LAT_COUNT: u16 = 721; + +/// Number of longitude columns in the ERA5 0.25 degree grid. `lon_idx` +/// ranges over `0..LON_COUNT` and is cyclic (wraps modulo `LON_COUNT`). +/// +/// `1440 / 64 = 22.5`: the last longitude tile (HEEL = 22) is ragged, +/// holding only the 32 columns `1408..1440` (plan §1.3 wrinkle 1). +pub const LON_COUNT: u16 = 1440; + +/// Length in bytes of the canonical `NodeGuid` key this module emits. +pub const KEY_LEN: usize = 16; + +/// The HEEL/HIP split point on each axis: `HEEL = idx >> TILE_SHIFT`, +/// `HIP = idx & TILE_MASK`. A 64-wide tile per axis. +const TILE_SHIFT: u16 = 6; + +/// `(1 << TILE_SHIFT) - 1` -- the low-bits mask recovering the HIP (within +/// tile) component of a grid index. +const TILE_MASK: u16 = (1 << TILE_SHIFT) - 1; + +/// Encodes a grid cell `(lat_idx, lon_idx)` into the canonical 16-byte key, +/// under the caller-supplied `classid`. +/// +/// `classid` is stored verbatim as little-endian bytes -- this module never +/// composes, mints, or bit-manipulates it (`D-WXS-0` is blocked; see the +/// module doc's byte-layout table). +/// +/// # Preconditions +/// +/// `lat_idx < LAT_COUNT` and `lon_idx < LON_COUNT`. Violating this is a +/// caller bug: in debug builds it trips a `debug_assert`; in release builds +/// the HEEL/HIP split still runs (silently producing a key that will not +/// round-trip through [`decode_key`], since the out-of-range guard lives on +/// the decode side, not here). Callers that need to construct or inspect an +/// out-of-range key directly (e.g. to test rejection) should build the +/// `[u8; 16]` by hand rather than calling this function with an +/// out-of-range index. +pub fn encode_key(classid: u32, lat_idx: u16, lon_idx: u16) -> [u8; KEY_LEN] { + debug_assert!( + lat_idx < LAT_COUNT, + "lat_idx {lat_idx} out of range (0..{LAT_COUNT})" + ); + debug_assert!( + lon_idx < LON_COUNT, + "lon_idx {lon_idx} out of range (0..{LON_COUNT})" + ); + + let mut key = [0u8; KEY_LEN]; + key[0..4].copy_from_slice(&classid.to_le_bytes()); + key[4] = (lat_idx >> TILE_SHIFT) as u8; + key[5] = (lon_idx >> TILE_SHIFT) as u8; + key[6] = (lat_idx & TILE_MASK) as u8; + key[7] = (lon_idx & TILE_MASK) as u8; + // bytes 8..10 (TWIG) stay 0 -- dormant, reserved. + // bytes 10..16 stay 0 -- V1-legacy tail; this crate mints nothing there. + key +} + +/// Decodes a canonical 16-byte key back into its grid cell `(lat_idx, +/// lon_idx)`, or `None` if the encoded cell falls outside the real grid +/// (`0..LAT_COUNT` x `0..LON_COUNT`). +/// +/// The bound check is deliberately on the *reconstructed* index +/// (`heel << TILE_SHIFT | hip`), not on the HEEL/HIP bytes individually -- +/// this is what makes an out-of-range cell (e.g. `lat_idx == LAT_COUNT`, +/// one past the last valid row) come back `None` instead of silently +/// wrapping or aliasing onto a real cell. `classid` and the TWIG/tail bytes +/// are not inspected; only the lat/lon axes are decoded. +pub fn decode_key(key: &[u8; KEY_LEN]) -> Option<(u16, u16)> { + let heel_lat = key[4] as u16; + let heel_lon = key[5] as u16; + let hip_lat = key[6] as u16; + let hip_lon = key[7] as u16; + + let lat_idx = (heel_lat << TILE_SHIFT) | hip_lat; + let lon_idx = (heel_lon << TILE_SHIFT) | hip_lon; + + if lat_idx >= LAT_COUNT || lon_idx >= LON_COUNT { + return None; + } + Some((lat_idx, lon_idx)) +} + +/// A rectangular, half-open, non-wrapping box of grid cells: +/// `lat_lo..lat_hi` and `lon_lo..lon_hi`. +/// +/// The only place a longitude wrap is resolved is [`box_ranges`] -- every +/// `CellBox` it hands back already excludes the wrap; a `CellBox` value is +/// never itself wrapping. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct CellBox { + /// Inclusive lower latitude index. + pub lat_lo: u16, + /// Exclusive upper latitude index. + pub lat_hi: u16, + /// Inclusive lower longitude index. + pub lon_lo: u16, + /// Exclusive upper longitude index. + pub lon_hi: u16, +} + +/// Splits a lat/lon box into the non-wrapping [`CellBox`] range(s) that +/// cover it (plan §1.3 wrinkle 2: "a box crossing the 0deg/360deg seam is +/// two ranges, never one"). +/// +/// `lat_lo..lat_hi` is passed through unchanged on every returned box -- +/// latitude never wraps (it has poles, not a seam). +/// +/// `lon_lo` and `lon_hi` are both grid indices in `0..=LON_COUNT`: +/// +/// * If `lon_lo < lon_hi`, the box does not cross the seam and this returns +/// **exactly one** `CellBox` covering `lon_lo..lon_hi`. +/// * If `lon_lo >= lon_hi`, the box wraps (the caller is expressing "from +/// `lon_lo` around through 0deg to `lon_hi`") and this returns **exactly +/// two**: `lon_lo..LON_COUNT`, then `0..lon_hi`. +/// +/// The return type is deliberate: a `Vec` (never an `Option`) so a +/// caller cannot silently drop the second range the way an optional-single- +/// range API would let them -- callers must handle however many ranges come +/// back, and `.len()` itself reports whether the box wrapped. +pub fn box_ranges(lat_lo: u16, lat_hi: u16, lon_lo: u16, lon_hi: u16) -> Vec { + if lon_lo < lon_hi { + vec![CellBox { + lat_lo, + lat_hi, + lon_lo, + lon_hi, + }] + } else { + vec![ + CellBox { + lat_lo, + lat_hi, + lon_lo, + lon_hi: LON_COUNT, + }, + CellBox { + lat_lo, + lat_hi, + lon_lo: 0, + lon_hi, + }, + ] + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashSet; + + /// Primary bar (B1): every one of the 1,038,240 real ERA5 0.25-degree + /// grid cells (`LAT_COUNT * LON_COUNT`) round-trips exactly through + /// `encode_key` -> `decode_key`, AND no two distinct cells produce the + /// same 16 bytes. This is **exhaustive, not sampled** -- ~1M iterations + /// is fast and this is the primary falsifier for the whole module: any + /// bug that makes two cells collide, or makes any single cell fail to + /// round-trip, fails this test. + #[test] + fn all_grid_cells_round_trip_and_are_collision_free() { + let classid = 0x1234_5678u32; + let total = LAT_COUNT as usize * LON_COUNT as usize; + assert_eq!(total, 1_038_240, "grid cell count drifted from the plan"); + + let mut seen: HashSet<[u8; KEY_LEN]> = HashSet::with_capacity(total); + for lat_idx in 0..LAT_COUNT { + for lon_idx in 0..LON_COUNT { + let key = encode_key(classid, lat_idx, lon_idx); + assert_eq!( + decode_key(&key), + Some((lat_idx, lon_idx)), + "round-trip failed for ({lat_idx}, {lon_idx})" + ); + assert!( + seen.insert(key), + "collision: ({lat_idx}, {lon_idx}) produced a key already seen" + ); + } + } + assert_eq!(seen.len(), total, "fewer distinct keys than grid cells"); + } + + /// Control that can lose: exercises the ragged last tile on BOTH axes + /// explicitly, using hardcoded bounds (704/721, 1408/1440) rather than + /// `LAT_COUNT`/`LON_COUNT`. If the exhaustive sweep above ever had its + /// bounds accidentally clamped down to the last *full* tile (704 lat + /// rows, 1408 lon columns -- e.g. a copy-pasted `& !TILE_MASK`), this + /// test would still catch it, because it does not share that bug's + /// source of truth. + #[test] + fn ragged_last_tile_cells_round_trip_on_both_axes() { + let classid = 0x0102_0304u32; + for lat_idx in 704..721u16 { + for lon_idx in 1408..1440u16 { + let key = encode_key(classid, lat_idx, lon_idx); + assert_eq!( + decode_key(&key), + Some((lat_idx, lon_idx)), + "ragged-tile cell ({lat_idx}, {lon_idx}) did not round-trip" + ); + } + } + } + + /// Stay-silent-twin partner of the ragged-tile test above, on the + /// rejection side: a cell exactly one past the grid on each axis must + /// be rejected by `decode_key`, not silently wrapped or aliased onto a + /// real cell. Keys are built by hand (not via `encode_key`, whose + /// precondition these indices violate) so the test exercises + /// `decode_key`'s own bound check in isolation. + #[test] + fn out_of_range_index_is_rejected_not_silently_wrapped() { + let classid = 0xAABB_CCDDu32; + + // lat_idx = 721 = 11*64 + 17: heel=11 (valid tile), hip=17 (one + // past that tile's 17 valid rows, 0..=16). + let mut lat_over = [0u8; KEY_LEN]; + lat_over[0..4].copy_from_slice(&classid.to_le_bytes()); + lat_over[4] = 11; // lat heel + lat_over[5] = 0; // lon heel (valid) + lat_over[6] = 17; // lat hip -> reconstructs to 721 + lat_over[7] = 0; // lon hip + assert_eq!( + decode_key(&lat_over), + None, + "lat_idx=721 (one past LAT_COUNT) must be rejected" + ); + + // lon_idx = 1440 = 22*64 + 32: heel=22 (valid tile), hip=32 (one + // past that tile's 32 valid columns, 0..=31). + let mut lon_over = [0u8; KEY_LEN]; + lon_over[0..4].copy_from_slice(&classid.to_le_bytes()); + lon_over[4] = 0; // lat heel (valid) + lon_over[5] = 22; // lon heel + lon_over[6] = 0; // lat hip + lon_over[7] = 32; // lon hip -> reconstructs to 1440 + assert_eq!( + decode_key(&lon_over), + None, + "lon_idx=1440 (one past LON_COUNT) must be rejected" + ); + + // Positive control: the true last valid cell on both axes must + // still decode fine -- proving the two rejections above are about + // being genuinely out of range, not a blanket "anything near the + // boundary fails" bug. + let last_valid = encode_key(classid, 720, 1439); + assert_eq!(decode_key(&last_valid), Some((720, 1439))); + } + + /// `box_ranges` can fire: a box crossing the 0deg/360deg seam reports + /// the wrap as exactly two ranges. + #[test] + fn box_ranges_seam_crossing_reports_exactly_two_ranges() { + // lon 1400..1440, wrapping around to lon 0..40. + let ranges = box_ranges(10, 20, 1400, 40); + assert_eq!(ranges.len(), 2, "seam-crossing box must report the wrap"); + assert_eq!( + ranges[0], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 1400, + lon_hi: LON_COUNT, + } + ); + assert_eq!( + ranges[1], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 0, + lon_hi: 40, + } + ); + } + + /// `box_ranges` can stay silent: an ordinary box entirely inside + /// `0..LON_COUNT` reports no wrap -- exactly one range, matching the + /// input bounds verbatim. Paired with the can-fire test above so + /// neither half of the wrap detector goes unchecked. + #[test] + fn box_ranges_non_wrapping_box_reports_exactly_one_range() { + let ranges = box_ranges(10, 20, 200, 400); + assert_eq!(ranges.len(), 1, "non-wrapping box must report no wrap"); + assert_eq!( + ranges[0], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 200, + lon_hi: 400, + } + ); + } +} diff --git a/probes/weather-p1/era5_variable_census.json b/probes/weather-p1/era5_variable_census.json new file mode 100644 index 000000000..13f41cc11 --- /dev/null +++ b/probes/weather-p1/era5_variable_census.json @@ -0,0 +1,75 @@ +{ + "coordinate_arrays_excluded": [ + "latitude", + "level", + "longitude", + "time" + ], + "counts": { + "levels": 13, + "pressure_level_fields": 91, + "pressure_level_variables": 7, + "static": 14, + "surface": 17, + "unclassified": 0 + }, + "derived": { + "bytes_per_timestep_512B_rows": 531578880, + "values_per_timestep": 126665280 + }, + "grid": { + "lat": 721, + "lon": 1440, + "n_cells": 1038240 + }, + "metadata_url": "https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr/.zmetadata", + "n_timesteps": 92044, + "pressure_level_variables": [ + "geopotential", + "specific_humidity", + "temperature", + "u_component_of_wind", + "v_component_of_wind", + "vertical_velocity", + "wind_speed" + ], + "probe": "era5_variable_census", + "static_variables": [ + "angle_of_sub_gridscale_orography", + "anisotropy_of_sub_gridscale_orography", + "geopotential_at_surface", + "high_vegetation_cover", + "lake_cover", + "lake_depth", + "land_sea_mask", + "low_vegetation_cover", + "slope_of_sub_gridscale_orography", + "soil_type", + "standard_deviation_of_filtered_subgrid_orography", + "standard_deviation_of_orography", + "type_of_high_vegetation", + "type_of_low_vegetation" + ], + "store_url": "https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr", + "surface_variables": [ + "10m_u_component_of_wind", + "10m_v_component_of_wind", + "10m_wind_speed", + "2m_temperature", + "mean_sea_level_pressure", + "sea_ice_cover", + "sea_surface_temperature", + "surface_pressure", + "toa_incident_solar_radiation", + "toa_incident_solar_radiation_12hr", + "toa_incident_solar_radiation_24hr", + "toa_incident_solar_radiation_6hr", + "total_cloud_cover", + "total_column_water_vapour", + "total_precipitation_12hr", + "total_precipitation_24hr", + "total_precipitation_6hr" + ], + "total_fields_per_cell": 122, + "unclassified_arrays": {} +} diff --git a/probes/weather-p1/era5_variable_census.py b/probes/weather-p1/era5_variable_census.py new file mode 100644 index 000000000..1dd9f0b62 --- /dev/null +++ b/probes/weather-p1/era5_variable_census.py @@ -0,0 +1,291 @@ +"""D-WXS-1a — the ERA5 WeatherBench2 Zarr variable census, as a committed, +re-runnable probe. + +`weather-soa-bake-v1.md` §0.6 asserts a variable census in prose (17 surface ++ 7 pressure-level x 13 levels = 91 + 14 static = 122 fields/cell, 92,044 +timesteps) with no artifact behind it. That is exactly the chat-only-figure +defect this arc has already caught three times (see the plan's §0 C1/⊘C3 +note: "a title match is not an existence check; list the bucket"). This +probe reads the store's own `.zmetadata` and emits the census as JSON, so +the numbers terminate at bytes fetched from the store rather than at a +sentence someone typed. + +Classification is by array shape rank, excluding the four coordinate +arrays (`time`, `level`, `latitude`, `longitude`): + + rank 3 [T, 721, 1440] -> time-varying SURFACE field + rank 4 [T, n_levels, 721, 1440] -> time-varying PRESSURE-LEVEL field + (contributes n_levels fields, one + per level) + rank 2 [721, 1440] -> STATIC field (no time axis) + +Anything that does not match one of those three shapes is reported under +`unclassified` rather than silently dropped or silently forced into a +bucket -- an unanticipated array shape is itself a finding. + +Run with `--selftest` to compare the fetched census against the constants +this plan already committed. A mismatch there means one of two things: +the store changed shape since the plan was written, or the plan's prose +was wrong. Either way the probe's own fetched answer wins -- that is the +point of terminating at the artifact instead of the prose. +""" + +import json +import pathlib +import sys +import urllib.error +import urllib.request + +STORE = ("https://storage.googleapis.com/weatherbench2/datasets/era5/" + "1959-2022-6h-1440x721.zarr") +METADATA_URL = STORE + "/.zmetadata" + +# The four coordinate arrays. Every other array in the store is a data +# variable (or an unanticipated shape, reported separately). +COORD_ARRAYS = {"time", "level", "latitude", "longitude"} + +HERE = pathlib.Path(__file__).parent +OUT = HERE / "era5_variable_census.json" + +# ---------------------------------------------------------------- fetch + +def _fetch_bytes(url, timeout=120): + """Fetch `url`, trying the default (proxied) opener first and falling + back to a proxy-bypassed opener on failure. + + This environment's outbound HTTPS goes through a pre-configured + agent proxy; a 403 or TLS error against a public GCS bucket is + usually the proxy, not the resource (see CLAUDE.md's own "GitHub + access matrix" finding, which generalizes past GitHub: verify with + the proxy bypassed before concluding a store is unreachable). + """ + try: + with urllib.request.urlopen(url, timeout=timeout) as resp: + return resp.read() + except Exception: # noqa: BLE001 -- broad on purpose, see docstring + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + with opener.open(url, timeout=timeout) as resp: + return resp.read() + + +def fetch_metadata(): + """Fetch and parse the store's consolidated `.zmetadata`. + + Returns the top-level `"metadata"` dict: keys are `"/.zarray"` + (plus `.zgroup` / `.zattrs` entries, which this probe ignores), values + carry `"shape"` and `"dtype"` among other zarr array-metadata fields. + """ + raw = _fetch_bytes(METADATA_URL) + doc = json.loads(raw) + return doc["metadata"] + + +# ------------------------------------------------------------ classify + +def array_entries(metadata): + """Yield (array_name, zarray_dict) for every `.zarray` entry in the + consolidated metadata, in the order the JSON gave them (not sorted -- + sorting happens once, at emission, on the classified name lists).""" + for key, val in metadata.items(): + if key.endswith("/.zarray"): + name = key[: -len("/.zarray")] + yield name, val + + +def classify(metadata): + """Classify every non-coordinate array by shape rank. + + Returns a dict with `surface`, `pressure_level` (name -> shape), + `static`, and `unclassified` (name -> shape, for anything that + doesn't match one of the three canonical shapes) plus the resolved + grid dims, timestep count, and level count pulled from the + coordinate arrays themselves (never hardcoded). + """ + coord_shapes = {} + for name, za in array_entries(metadata): + if name in COORD_ARRAYS: + coord_shapes[name] = za["shape"] + + missing = COORD_ARRAYS - set(coord_shapes) + if missing: + raise RuntimeError( + f"store is missing expected coordinate array(s): {sorted(missing)} " + "-- cannot classify without a time/level/lat/lon reference") + + n_time = coord_shapes["time"][0] + n_levels = coord_shapes["level"][0] + n_lat = coord_shapes["latitude"][0] + n_lon = coord_shapes["longitude"][0] + grid_shape = [n_lat, n_lon] + + surface, pressure_level, static, unclassified = {}, {}, {}, {} + + for name, za in array_entries(metadata): + if name in COORD_ARRAYS: + continue + shape = za["shape"] + rank = len(shape) + if rank == 2 and shape == grid_shape: + static[name] = shape + elif rank == 3 and shape[0] == n_time and shape[1:] == grid_shape: + surface[name] = shape + elif (rank == 4 and shape[0] == n_time and shape[1] == n_levels + and shape[2:] == grid_shape): + pressure_level[name] = shape + else: + unclassified[name] = shape + + return { + "surface": surface, + "pressure_level": pressure_level, + "static": static, + "unclassified": unclassified, + "n_time": n_time, + "n_levels": n_levels, + "n_lat": n_lat, + "n_lon": n_lon, + } + + +# --------------------------------------------------------------- report + +def build_report(metadata): + """Assemble the full committed-JSON census from a classified store.""" + c = classify(metadata) + + surface_names = sorted(c["surface"]) + pressure_names = sorted(c["pressure_level"]) + static_names = sorted(c["static"]) + unclassified_names = sorted(c["unclassified"]) + coord_names = sorted(COORD_ARRAYS) + + n_surface = len(surface_names) + n_pressure_vars = len(pressure_names) + n_levels = c["n_levels"] + n_pressure_fields = n_pressure_vars * n_levels + n_static = len(static_names) + total_fields_per_cell = n_surface + n_pressure_fields + n_static + + n_lat, n_lon = c["n_lat"], c["n_lon"] + n_cells = n_lat * n_lon + n_time = c["n_time"] + + values_per_timestep = n_cells * total_fields_per_cell + bytes_per_timestep_512b_rows = n_cells * 512 + + return { + "probe": "era5_variable_census", + "store_url": STORE, + "metadata_url": METADATA_URL, + "coordinate_arrays_excluded": coord_names, + "surface_variables": surface_names, + "pressure_level_variables": pressure_names, + "static_variables": static_names, + "unclassified_arrays": {n: c["unclassified"][n] for n in unclassified_names}, + "counts": { + "surface": n_surface, + "pressure_level_variables": n_pressure_vars, + "levels": n_levels, + "pressure_level_fields": n_pressure_fields, + "static": n_static, + "unclassified": len(unclassified_names), + }, + "total_fields_per_cell": total_fields_per_cell, + "grid": {"lat": n_lat, "lon": n_lon, "n_cells": n_cells}, + "n_timesteps": n_time, + "derived": { + "values_per_timestep": values_per_timestep, + "bytes_per_timestep_512B_rows": bytes_per_timestep_512b_rows, + }, + } + + +# --------------------------------------------------------------- selftest + +# The constants `weather-soa-bake-v1.md` §0.6 already committed in prose. +# A mismatch means either the store changed since the plan was written, or +# the plan's prose figure was wrong -- either way this probe's fetched +# answer is the one that wins, because it terminates at the artifact. +EXPECTED = { + "surface": 17, + "pressure_level_variables": 7, + "levels": 13, + "pressure_level_fields": 91, + "static": 14, + "total_fields_per_cell": 122, + "n_cells": 1_038_240, + "n_timesteps": 92_044, + "values_per_timestep": 126_665_280, + "bytes_per_timestep_512B_rows": 531_578_880, +} + + +def selftest(report): + """Compare the fetched report against EXPECTED. Fails loudly (exit 1) + and prints every mismatch on disagreement -- never adjusts EXPECTED + to make a run pass.""" + got = { + "surface": report["counts"]["surface"], + "pressure_level_variables": report["counts"]["pressure_level_variables"], + "levels": report["counts"]["levels"], + "pressure_level_fields": report["counts"]["pressure_level_fields"], + "static": report["counts"]["static"], + "total_fields_per_cell": report["total_fields_per_cell"], + "n_cells": report["grid"]["n_cells"], + "n_timesteps": report["n_timesteps"], + "values_per_timestep": report["derived"]["values_per_timestep"], + "bytes_per_timestep_512B_rows": + report["derived"]["bytes_per_timestep_512B_rows"], + } + mismatches = {k: (EXPECTED[k], got[k]) for k in EXPECTED if EXPECTED[k] != got[k]} + return mismatches, got + + +# -------------------------------------------------------------------- main + +def main(): + run_selftest = "--selftest" in sys.argv + + metadata = fetch_metadata() + report = build_report(metadata) + + OUT.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n") + + print(f"store: {STORE}") + print(f"metadata: {METADATA_URL}") + print(f"surface variables: {report['counts']['surface']}") + print(f"pressure-level variables: {report['counts']['pressure_level_variables']}" + f" x {report['counts']['levels']} levels" + f" = {report['counts']['pressure_level_fields']} fields") + print(f"static variables: {report['counts']['static']}") + print(f"unclassified arrays: {report['counts']['unclassified']}" + + (f" {sorted(report['unclassified_arrays'])}" + if report['unclassified_arrays'] else "")) + print(f"total fields/cell: {report['total_fields_per_cell']}") + print(f"grid: {report['grid']['lat']} x {report['grid']['lon']}" + f" = {report['grid']['n_cells']} cells") + print(f"timesteps: {report['n_timesteps']}") + print(f"values/timestep: {report['derived']['values_per_timestep']}") + print(f"bytes/timestep @512B rows: " + f"{report['derived']['bytes_per_timestep_512B_rows']}") + print(f"\nwrote {OUT}") + + if run_selftest: + mismatches, got = selftest(report) + if mismatches: + print("\nSELFTEST: FAIL -- fetched census disagrees with the plan's " + "committed constants (weather-soa-bake-v1.md §0.6):") + for k, (exp, act) in sorted(mismatches.items()): + print(f" {k:32s} expected={exp!r:>14} got={act!r:>14}") + print("\nThis means either the store changed shape since the plan " + "was written, or the plan's prose figure was wrong. The " + "fetched answer above is authoritative; do not edit EXPECTED " + "to silence this without updating the plan too.") + sys.exit(1) + else: + print(f"\nSELFTEST: PASS -- all {len(EXPECTED)} committed constants " + "match the fetched census.") + + +if __name__ == "__main__": + main() From dc5166a5e8cd0d918504711199ef28bd07feca72 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 09:33:08 +0000 Subject: [PATCH 8/8] D-WXS-1 + D-WXS-3 SHIPPED: the ClassView-side manifest and the canonical floor The W0/W1 fleet's second half. Both gated by the orchestrator rather than taken from the workers' reports. D-WXS-3 -- the shared canonical floor. Calibration (0.4-99.6 percentile window, a floor_version stamp, occupancy, saturation) is genuinely new; quantize and bucket_center are a documented re-expression of helix::quantize::RollingFloor with line references, because this crate is deliberately zero-dep and cannot take helix's mandatory ndarray git dep through an optional path edge. 7/7. Bar B2 disable-verified: widening the "narrow" control floor kills only the control while the stay-silent twin holds. The worker chose a hand-rolled FNV-1a over DefaultHasher specifically because the latter is randomly seeded per process -- which would silently break the whole "same sample => same version" contract the stamp exists to provide. That is a real trap and the suite does not cover it, so it was verified separately: floor_version is byte-identical across three separate process invocations. The epsilon (0.03) is derived from the percentile construction -- the window trims ~0.4% per end and each rim bucket collects its own ~1/256 interior share, so ~1.57% expected, leaving ~2x margin -- not tuned until it passed. D-WXS-1 -- the field manifest. This deliverable exists to satisfy one rule: labels and positions come from the ClassView, NEVER from a slot in the payload (le-contract section 2). So the (facet, pair, byte) -> (variable, level, unit, floor) mapping is a committed TSV the bake reads, embedded via include_str!, parsed by hand -- no serde, no TSV crate, still zero deps. 22 rows: F0 5 pairs, F1 and F2 3 pairs each, and reserved slots emit NO row, which is what makes them dormant and expandable without a layout change. 13/13. Collision guard disable-verified: removed, only colliding_entries_are_rejected fails while BOTH stay-silent twins stay green -- distinct lo/hi bytes on one pair, and a byte-different-but-semantically-identical row reordering that must resolve every lookup identically. A validator that rejects everything carries exactly as much information as one that accepts everything. Bar B0's end-to-end half -- mutating an entry must change the bytes the bake writes -- is DEFERRED and says so in the module doc rather than being faked. The bake is D-WXS-4 and does not exist. Board: AGENT_LOG prepended with the consolidated 4-worker wave entry (the orchestrator is the sole writer; the consolidation itself is not an agent run and gets no entry of its own). STATUS_BOARD D-WXS-1 and D-WXS-3 rows moved from Queued to SHIPPED -- a Status-field edit, the one mutation the append-only rule sanctions. Gates: cargo test --manifest-path crates/weather-poc/Cargo.toml 25/25; clippy --all-targets -D warnings clean; fmt clean. All in the one shared target/. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/AGENT_LOG.md | 52 ++ .claude/board/STATUS_BOARD.md | 4 +- crates/weather-poc/data/field_manifest_v1.tsv | 43 ++ crates/weather-poc/src/floor.rs | 469 ++++++++++++- crates/weather-poc/src/manifest.rs | 616 +++++++++++++++++- 5 files changed, 1180 insertions(+), 4 deletions(-) create mode 100644 crates/weather-poc/data/field_manifest_v1.tsv diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 95aa64e42..5f1158146 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,55 @@ +## 2026-08-13 — weather-poc W0/W1 fleet (4 Sonnet workers, disjoint files; orchestrator-consolidated) + +4/4 completed, 0 errors. Plan `.claude/plans/weather-soa-bake-v1.md`; the first +grindwork wave on the crate `weather-substrate-poc-v2` named and never created. +Files were disjoint by construction (`key.rs` / `floor.rs` / +`manifest.rs`+`data/*.tsv` / `probes/weather-p1/era5_variable_census.*`); the +orchestrator owned `lib.rs`, both manifests, the root exclude entry and every +board file. Guardrails §1 pasted verbatim into all four briefs; no worker ran +cargo (rule 7) and all three Rust workers reported "not compiled, not run" +rather than claiming green. + +**D-WXS-1a census (Python, the one worker allowed to run).** Reads the store's +`.zmetadata`, emits the JSON, `--selftest` asserts all 10 constants the plan +carries. PASS, and re-run independently by the orchestrator. Guard +disable-verified twice — the worker broke a constant in a throwaway copy, and +the orchestrator repeated it against the committed file (exit 1, correct +"the fetched answer is authoritative" message). 0 arrays failed to classify. + +**D-WXS-2 key codec.** 5/5. Exhaustive 1,038,240-cell round-trip, collision- +checked. Bar B1 disable-verified by the ORCHESTRATOR, not asserted by the +worker: zeroing the HIP lat byte kills 3 tests (collision, ragged, +out-of-range); removing the seam split kills the wrap twin while the non-wrap +twin stays green — the pair discriminates in both directions. + +**D-WXS-3 floor.** 7/7. Bar B2 disable-verified: widening the "narrow" control +floor kills only the control, the stay-silent twin holds. The worker chose a +hand-rolled FNV-1a over `DefaultHasher` *because the latter is randomly seeded +per process* — a real trap, and one the suite does not cover, so the +orchestrator verified it separately: `floor_version` is byte-identical across +three separate process invocations. ε = 0.03 derived from the percentile +construction, not tuned post-hoc. + +**D-WXS-1 manifest.** 13/13, TSV 22 rows (F0 10 / F1 6 / F2 6, reserved slots +emit NO row — exactly plan §2.5). Collision guard disable-verified: removed → +only `colliding_entries_are_rejected` fails while BOTH stay-silent twins +(distinct lo/hi on one pair; row reordering) stay green. Bar B0's end-to-end +half is DEFERRED and said so in the module doc — the bake does not exist yet. + +**The wave's real find is a plan defect, not a code defect.** The key worker +noticed during implementation that §1.2 assigns one WHOLE byte per axis while +OGAR's cascade doctrine — the passage §1.1 leans on — specifies the axis bytes +nibble-interleaved (Morton). The plan deviated from the canon it cites and did +not say so. Recorded as plan §1.3a + board `D-WXS-2a` with a pre-registered +comparison, NOT silently rewritten: the prefix-scan claim holds under both +layouts, and rewriting a spec the moment a worker surfaces a consequence is how +a correction starts building. + +Gates (orchestrator, one shared `target/`): `cargo test --manifest-path +crates/weather-poc/Cargo.toml` **25/25**; `cargo clippy --all-targets -D +warnings` clean; `cargo fmt` clean. Commits `010da851` (scaffold), `a8a501bc` +(census + key), this one (floor + manifest). + ## 2026-08-11 — Workflow `wf_99d677e6-b45` — eval-plan verify/attack (13 agents; orchestrator-consolidated) 13/13 completed, 0 errors, 0 empty; ~4.42 M subagent tokens, 165 tool calls, diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 3eaef02b1..c302ba1bb 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -9,11 +9,11 @@ with a control that can lose and a stay-silent twin. | D-id | Deliverable | Wave | Status | Feeds | |---|---|---|---|---| | D-WXS-0 | classid mint for the weather-cell + statics classes (OGAR-side; `0x0F = Geo` exists, appid/classview open) | 0 | **Blocked (operator/OGAR)** | the bake must REFUSE to write under `0x0000_0000` | -| D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | Queued | slot purity §2; bar B0 | +| D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | **SHIPPED 2026-08-13** — `data/field_manifest_v1.tsv` (22 rows = F0 5 pairs + F1/F2 3 pairs each, reserved slots emit NO row) + `manifest.rs`, 13/13; collision guard **disable-verified** (removed → only `colliding_entries_are_rejected` fails, BOTH stay-silent twins stay green). Bar B0's end-to-end half (mutating an entry changes written bytes) DEFERRED — the bake does not exist yet | slot purity §2; bar B0 | | D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | | D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | | D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | -| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | Queued | bar B2 | +| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | | D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | | D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | | D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions | diff --git a/crates/weather-poc/data/field_manifest_v1.tsv b/crates/weather-poc/data/field_manifest_v1.tsv new file mode 100644 index 000000000..cc1dac4c3 --- /dev/null +++ b/crates/weather-poc/data/field_manifest_v1.tsv @@ -0,0 +1,43 @@ +# field_manifest_v1.tsv -- the ClassView-side field manifest for weather-poc +# W1 (plan .claude/plans/weather-soa-bake-v1.md §2.5/§2.6/D-WXS-1). +# +# One row per OCCUPIED byte of the L4 `6x(8:8)` facet layout +# (.claude/v3/soa_layout/le-contract.md §2/§3). Reserved-zero slots are NOT +# rows (plan §2.5) -- their absence IS their meaning: dormant, expandable +# later without a layout change (RESERVE, DON'T RECLAIM). +# +# lo/hi convention: within a pair written "A : B" in the plan §2.5 table, +# A is the LOW byte and B is the HIGH byte -- the pair's own left-to-right +# order. Kept consistent across every row below. +# +# facet: 0 = F0 (surface), 1 = F1 (850 hPa), 2 = F2 (500 hPa) -- plan §2.5. +# variable: the ERA5 variable name exactly as it appears in the Zarr store +# (verified against a live .zmetadata read, plan §0.6). +# level_hpa: the pressure level in hPa, or the literal token `surface`. +# floor_id: a stable identifier for the calibration floor this field uses; +# two rows sharing a floor_id share the same [lo, hi] calibration (none do +# in this committed manifest -- each field here is calibrated on its own +# physical range, plan §2.3/D-WXS-3). +facet pair byte variable level_hpa unit floor_id +0 0 lo mean_sea_level_pressure surface Pa msl +0 0 hi surface_pressure surface Pa sp +0 1 lo 10m_u_component_of_wind surface m s-1 u10 +0 1 hi 10m_v_component_of_wind surface m s-1 v10 +0 2 lo 10m_wind_speed surface m s-1 wspd10 +0 2 hi 2m_temperature surface K t2m +0 3 lo sea_surface_temperature surface K sst +0 3 hi total_column_water_vapour surface kg m-2 tcwv +0 4 lo total_cloud_cover surface 1 tcc +0 4 hi sea_ice_cover surface 1 sic +1 0 lo u_component_of_wind 850 m s-1 u850 +1 0 hi v_component_of_wind 850 m s-1 v850 +1 1 lo temperature 850 K t850 +1 1 hi specific_humidity 850 kg kg-1 q850 +1 2 lo geopotential 850 m2 s-2 z850 +1 2 hi vertical_velocity 850 Pa s-1 w850 +2 0 lo u_component_of_wind 500 m s-1 u500 +2 0 hi v_component_of_wind 500 m s-1 v500 +2 1 lo temperature 500 K t500 +2 1 hi specific_humidity 500 kg kg-1 q500 +2 2 lo geopotential 500 m2 s-2 z500 +2 2 hi vertical_velocity 500 Pa s-1 w500 diff --git a/crates/weather-poc/src/floor.rs b/crates/weather-poc/src/floor.rs index 96c93dcae..03347f72c 100644 --- a/crates/weather-poc/src/floor.rs +++ b/crates/weather-poc/src/floor.rs @@ -1 +1,468 @@ -//! Placeholder — owned by the floor worker (plan §6.2). Filled in W1. +//! `D-WXS-3` — the shared canonical floor calibration + the linear 256-level +//! quantiser (`.claude/plans/weather-soa-bake-v1.md` §4 W1, bar B2). +//! +//! Two halves, one genuinely new, one a documented re-expression: +//! +//! 1. **Calibration (new work).** [`calibrate`] derives a floor window +//! `[lo, hi]` from a sample of `f64` values at the robust +//! **0.4th – 99.6th percentile** ([`LO_PERCENTILE`] / [`HI_PERCENTILE`]) +//! — a window that deliberately excludes the extreme tails so a single +//! outlier sample cannot collapse `[lo, hi]` toward a degenerate span — +//! and stamps the result with a [`CalibratedFloor::floor_version`] +//! identifying the calibration epoch that produced it. +//! 2. **The linear floor (a re-expression, not a re-derivation).** +//! [`CalibratedFloor::quantize`] and [`CalibratedFloor::bucket_center`] +//! are byte-for-byte the same arithmetic as +//! `helix::quantize::RollingFloor::{quantize, bucket_center}` +//! (`crates/helix/src/quantize.rs:99-108`, `:248-250`): 256 uniform +//! levels over `[lo, hi]`, clip-saturating outside the window, decoding +//! to the bucket centre. This crate is zero-dep BY CONSTRUCTION (see +//! `weather-poc/Cargo.toml`'s ZERO-DEP-BY-CONSTRUCTION note — an +//! *optional path* dep on `helix` would still be read at manifest +//! resolution and break a clean checkout), so the formula is +//! re-expressed here rather than imported, and the two implementations +//! must be kept in lockstep by inspection, not by a shared dependency +//! edge — the same shape as `D-WXS-12` (`jc` ↔ `ndarray::hpc` +//! agreement): parity is a measured comparison, not a dependency. +//! +//! [`CalibratedFloor::occupancy`] **is** the histogram — no separate +//! instrument is kept, mirroring the property +//! `helix::quantize::RollingFloor`'s own module docs state +//! (`crates/helix/src/quantize.rs:5-9`). + +/// Number of quantisation buckets. Mirrors `helix::constants::PALETTE_SIZE` +/// (`= 256`), duplicated here rather than imported — see the module docs. +const BUCKETS: usize = 256; + +/// Lower percentile of the robust calibration window (0.4th percentile). +/// +/// Chosen — per the deliverable spec — to exclude the extreme tail so a +/// single outlier sample cannot collapse `[lo, hi]` to a degenerate or +/// near-degenerate span. See [`HI_PERCENTILE`] for the upper bound and the +/// module docs for the full window. +pub const LO_PERCENTILE: f64 = 0.4; + +/// Upper percentile of the robust calibration window (99.6th percentile). +/// See [`LO_PERCENTILE`]. +pub const HI_PERCENTILE: f64 = 99.6; + +/// A calibrated, versioned linear 256-bucket floor over `[lo, hi]`. +/// +/// Produced only by [`calibrate`] — there is no public constructor that +/// bypasses calibration, because [`Self::floor_version`] is a promise about +/// *how* `(lo, hi)` was derived, and a hand-built floor would break that +/// promise silently. +/// +/// The quantise/dequantise arithmetic ([`Self::quantize`], +/// [`Self::bucket_center`]) is the re-expressed half of `D-WXS-3` — see the +/// module docs for the exact `helix::quantize::RollingFloor` line +/// references it mirrors. +#[derive(Debug, Clone)] +pub struct CalibratedFloor { + lo: f64, + hi: f64, + floor_version: u64, + occupancy: [u32; BUCKETS], + samples: u64, +} + +impl CalibratedFloor { + /// **COMPUTE — `&self`.** Map `value` to a bucket in `0..=255`. + /// + /// Identical formula to `helix::quantize::RollingFloor::quantize` + /// (`crates/helix/src/quantize.rs:99-108`): + /// + /// - Returns `0` for degenerate bounds (`hi <= lo`). + /// - Saturates to bucket `0` for `value <= lo` and to bucket `255` for + /// `value >= hi`. + /// - Otherwise `idx = floor(((value - lo) / (hi - lo)) * 256)`, clamped + /// to `[0, 255]`. + pub fn quantize(&self, value: f64) -> u8 { + if self.hi <= self.lo { + return 0; + } + let t = (value - self.lo) / (self.hi - self.lo); + let idx = (t * 256.0).floor(); + let idx = idx.clamp(0.0, 255.0); + idx as u8 + } + + /// **COMPUTE — `&self`.** The representative centre value of bucket `b`. + /// + /// Identical formula to `helix::quantize::RollingFloor::bucket_center` + /// (`crates/helix/src/quantize.rs:248-250`): + /// `lo + ((b + 0.5) / 256) * (hi - lo)`. + pub fn bucket_center(&self, b: u8) -> f64 { + self.lo + ((b as f64 + 0.5) / BUCKETS as f64) * (self.hi - self.lo) + } + + /// Decode bucket `b`, but only when `floor_version` matches this + /// floor's own [`Self::floor_version`]. + /// + /// Returns `None` on a version mismatch instead of silently returning a + /// bucket centre computed from the WRONG `[lo, hi]` window. + /// `helix::quantize::RollingFloor`'s module docs state the same + /// contract for its own version stamp (`crates/helix/src/quantize.rs: + /// 20-26`): "same value → same `u8`" holds only within a stable floor + /// version, so decoding across a version boundary without checking is + /// exactly the silent-mis-dequantisation failure this method closes. + pub fn decode(&self, b: u8, floor_version: u64) -> Option { + if floor_version != self.floor_version { + return None; + } + Some(self.bucket_center(b)) + } + + /// The calibration-epoch stamp. + /// + /// A deterministic FNV-1a-64 hash (see the private `fnv1a_64` helper in + /// this module) of: `lo`, `hi` (the calibration OUTPUT), the + /// calibration sample's finite-value count, and [`LO_PERCENTILE`] / + /// [`HI_PERCENTILE`] (the calibration PARAMETERS). Because `(lo, hi)` + /// is itself a deterministic function of the input sample and the + /// percentiles, this satisfies "same sample + same percentiles ⇒ same + /// version": re-calibrating on a byte-identical sample with the same + /// percentile window reproduces the same `lo`, `hi`, and count, hence + /// the same hash. + /// + /// It does NOT guarantee the converse — two different samples that + /// happen to land on byte-identical `(lo, hi, count)` would share a + /// version — but that is correct rather than a gap: `(lo, hi)` is the + /// only state [`Self::quantize`] and [`Self::bucket_center`] ever + /// consult, so two floors with the same `(lo, hi)` really are the same + /// calibration epoch as far as decoding is concerned. + /// + /// `std::collections::hash_map::DefaultHasher` (SipHash) was + /// deliberately NOT used here: its seed is randomised per process, so + /// two runs of the identical calibration would produce two different + /// stamps — exactly the property this stamp must not have. + pub fn floor_version(&self) -> u64 { + self.floor_version + } + + /// Current `(lo, hi)` calibration bounds. + pub fn bounds(&self) -> (f64, f64) { + (self.lo, self.hi) + } + + /// Per-bucket occupancy counts over the calibration sample, length 256. + /// This **is** the histogram — no separate instrument is kept. See the + /// module docs. + pub fn occupancy(&self) -> &[u32; BUCKETS] { + &self.occupancy + } + + /// Number of finite values the calibration sample contributed to + /// [`Self::occupancy`] (non-finite values are excluded — see + /// [`calibrate`]). + pub fn samples(&self) -> u64 { + self.samples + } + + /// **COMPUTE — `&self`.** Fraction of the CALIBRATION SAMPLE landing in + /// the two rim buckets (`0` and `255`), read straight off + /// [`Self::occupancy`]. Returns `0.0` when [`Self::samples`] is `0`. + pub fn saturation(&self) -> f64 { + if self.samples == 0 { + return 0.0; + } + let rim = self.occupancy[0] as u64 + self.occupancy[BUCKETS - 1] as u64; + rim as f64 / self.samples as f64 + } + + /// **COMPUTE — `&self`.** Fraction of an ARBITRARY external population + /// `values` that lands in the two rim buckets under this floor. + /// + /// Unlike [`Self::saturation`] (which scores the calibration sample's + /// own recorded [`Self::occupancy`]), this re-quantises `values` + /// against this floor's `[lo, hi]` on the fly, without mutating + /// anything. Both directions bar B2 needs are "this floor scored + /// against a population it was not calibrated on": a narrow floor + /// scored globally, and a global floor scored on one narrow box. + /// Returns `0.0` on an empty `values` slice. + pub fn saturation_of(&self, values: &[f64]) -> f64 { + if values.is_empty() { + return 0.0; + } + let rim = values + .iter() + .filter(|&&v| { + let b = self.quantize(v); + b == 0 || b == (BUCKETS - 1) as u8 + }) + .count(); + rim as f64 / values.len() as f64 + } +} + +/// Deterministic FNV-1a 64-bit hash over raw bytes. See +/// [`CalibratedFloor::floor_version`] for why this, and not +/// `std::collections::hash_map::DefaultHasher`, backs the version stamp. +fn fnv1a_64(bytes: &[u8]) -> u64 { + const FNV_OFFSET_BASIS: u64 = 0xcbf2_9ce4_8422_2325; + const FNV_PRIME: u64 = 0x0000_0100_0000_01b3; + let mut hash = FNV_OFFSET_BASIS; + for &byte in bytes { + hash ^= byte as u64; + hash = hash.wrapping_mul(FNV_PRIME); + } + hash +} + +/// Linear-interpolated percentile (the "R-7" / `PERCENTILE.INC` method) +/// over an already-sorted, non-empty, all-finite slice. +/// +/// `p` is expected in `[0, 100]`. `rank = (p / 100) * (n - 1)`; +/// interpolates between the two adjacent order statistics when `rank` is +/// not an integer. +fn percentile(sorted: &[f64], p: f64) -> f64 { + let n = sorted.len(); + if n == 1 { + return sorted[0]; + } + let rank = (p / 100.0) * (n as f64 - 1.0); + let lo_idx = (rank.floor() as usize).min(n - 1); + let hi_idx = (rank.ceil() as usize).min(n - 1); + if lo_idx == hi_idx { + sorted[lo_idx] + } else { + let frac = rank - lo_idx as f64; + sorted[lo_idx] * (1.0 - frac) + sorted[hi_idx] * frac + } +} + +/// Calibrate a [`CalibratedFloor`] from a sample of `f64` values. +/// +/// Derives `[lo, hi]` at the [`LO_PERCENTILE`]–[`HI_PERCENTILE`] robust +/// window (see the module docs), then re-scans the sample under the +/// resulting floor to build [`CalibratedFloor::occupancy`] and +/// [`CalibratedFloor::samples`] — the "genuinely new" half of `D-WXS-3`; +/// [`CalibratedFloor::quantize`] / [`CalibratedFloor::bucket_center`] are +/// the re-expressed half. +/// +/// Non-finite (`NaN` / `±inf`) values in `sample` are excluded from both +/// the percentile computation and the occupancy scan — they carry no +/// ordering information a percentile window can use. +/// +/// Returns `None` when `sample` contains no finite values (there is +/// nothing to calibrate a window from). +pub fn calibrate(sample: &[f64]) -> Option { + let mut finite: Vec = sample.iter().copied().filter(|v| v.is_finite()).collect(); + if finite.is_empty() { + return None; + } + finite.sort_by(|a, b| a.total_cmp(b)); + + let lo = percentile(&finite, LO_PERCENTILE); + let hi = percentile(&finite, HI_PERCENTILE); + + let mut version_bytes: Vec = Vec::with_capacity(8 * 5); + version_bytes.extend_from_slice(&lo.to_le_bytes()); + version_bytes.extend_from_slice(&hi.to_le_bytes()); + version_bytes.extend_from_slice(&(finite.len() as u64).to_le_bytes()); + version_bytes.extend_from_slice(&LO_PERCENTILE.to_le_bytes()); + version_bytes.extend_from_slice(&HI_PERCENTILE.to_le_bytes()); + let floor_version = fnv1a_64(&version_bytes); + + let mut floor = CalibratedFloor { + lo, + hi, + floor_version, + occupancy: [0u32; BUCKETS], + samples: 0, + }; + + for &v in &finite { + let b = floor.quantize(v); + floor.occupancy[b as usize] = floor.occupancy[b as usize].saturating_add(1); + floor.samples = floor.samples.saturating_add(1); + } + + Some(floor) +} + +#[cfg(test)] +mod tests { + use super::*; + + // ── fixture helper ─────────────────────────────────────────────────── + + /// Evenly spaced points spanning `[lo, hi]` inclusive — deterministic, + /// no RNG, so every calibration and every measured saturation number in + /// these tests is exactly reproducible. + fn linspace(lo: f64, hi: f64, n: usize) -> Vec { + assert!(n >= 2, "linspace needs at least 2 points"); + (0..n) + .map(|i| lo + (hi - lo) * i as f64 / (n as f64 - 1.0)) + .collect() + } + + // ── B2 primary: well-covered sample, zero empty buckets, low saturation ─ + + #[test] + fn primary_well_covered_sample_has_zero_empty_buckets_and_low_saturation() { + // Pre-registered epsilon, worked out from the construction BEFORE + // running: the 0.4/99.6 percentile window trims ~0.4% of the + // 102_400-point evenly-spaced fixture off each end (~409-410 + // points per side), and each rim bucket (0 and 255) ALSO collects + // its own ~1/256 share of the interior mass (~397 points) because + // the trimmed points and the interior points nearest lo/hi both + // land in the same rim bucket. That puts expected rim occupancy + // near (409 + 397 + 410 + 397) / 102_400 ≈ 1.57%. epsilon = 0.03 + // leaves roughly 2x margin above that expectation without being so + // loose that a real quantisation bug (e.g. an inverted saturation + // branch, which would push saturation toward 1.0) would slip past. + const EPSILON: f64 = 0.03; + + let sample = linspace(-1000.0, 1000.0, 102_400); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + + let empty_buckets = floor.occupancy().iter().filter(|&&c| c == 0).count(); + assert_eq!( + empty_buckets, 0, + "expected every one of the 256 buckets occupied, found {empty_buckets} empty" + ); + + let sat = floor.saturation(); + assert!( + sat < EPSILON, + "saturation {sat} did not clear the pre-registered epsilon {EPSILON}" + ); + } + + // ── B2 control that can lose: narrow floor applied globally saturates ── + + #[test] + fn control_narrow_floor_applied_globally_shows_high_saturation() { + // This is `GEO-DEGENERATE` re-homed (plan §4 bar B2): a floor + // calibrated on a narrow slice, applied to a much wider + // independent population, must show HIGH saturation. Known + // losable — the arc measured 72-97% on real data for exactly this + // construction. + let narrow_sample = linspace(-10.0, 10.0, 2_000); + let narrow_floor = calibrate(&narrow_sample).expect("non-empty sample calibrates"); + + // Anti-vacuity: confirm the fixture really IS narrow before + // scoring it — otherwise a passing assertion below would prove + // nothing about the narrow-vs-wide mechanism. + let (n_lo, n_hi) = narrow_floor.bounds(); + assert!( + n_hi - n_lo < 25.0, + "fixture is not actually narrow: bounds [{n_lo}, {n_hi}]" + ); + + let wide_population = linspace(-1000.0, 1000.0, 102_400); + let sat = narrow_floor.saturation_of(&wide_population); + assert!( + sat > 0.9, + "expected high saturation applying a narrow floor globally, got {sat}" + ); + } + + // ── B2 stay-silent twin: global floor on a non-trivial sub-window ────── + + #[test] + fn stay_silent_global_floor_on_a_narrow_sub_window_does_not_saturate() { + // The global floor scored against a non-trivial sub-window of the + // SAME population it was calibrated on must NOT show high + // saturation. The sub-window here is the population's own upper + // third — real values, not an empty set — chosen specifically + // because it brushes against (without mostly exceeding) the + // calibration window's own trimmed edge, which is a genuinely + // testable case rather than a trivially-safe interior slice. + let wide_population = linspace(-1000.0, 1000.0, 102_400); + let global_floor = calibrate(&wide_population).expect("non-empty sample calibrates"); + + let n = wide_population.len(); + let sub_window = &wide_population[(2 * n / 3)..n]; + assert!( + sub_window.len() > 1_000, + "sub-window must be non-trivial, got {} points", + sub_window.len() + ); + + let sat = global_floor.saturation_of(sub_window); + assert!( + sat < 0.1, + "global floor should not saturate heavily on an interior-adjacent sub-window, got {sat}" + ); + } + + // ── round-trip bound ───────────────────────────────────────────────── + + #[test] + fn round_trip_bucket_center_is_within_half_bucket_of_source_value() { + let sample = linspace(-500.0, 1500.0, 50_000); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + let (lo, hi) = floor.bounds(); + assert!( + hi > lo, + "calibration must be non-degenerate for this fixture" + ); + + let half_bucket = (hi - lo) / (2.0 * BUCKETS as f64); + let probe_count = 5_000usize; + for i in 0..probe_count { + // Offset by half a probe-step so we never land exactly on a + // window endpoint, which would legitimately saturate. + let t = (i as f64 + 0.5) / probe_count as f64; + let v = lo + t * (hi - lo); + let b = floor.quantize(v); + let center = floor.bucket_center(b); + let delta = (center - v).abs(); + assert!( + delta <= half_bucket + 1e-9, + "round-trip error {delta} exceeds the ±half-bucket bound {half_bucket} at v={v}" + ); + } + } + + // ── version stamp is load-bearing ─────────────────────────────────── + + #[test] + fn version_stamp_mismatch_is_detected_and_matching_version_decodes() { + let sample = linspace(-50.0, 50.0, 5_000); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + + let real_version = floor.floor_version(); + let bogus_version = real_version.wrapping_add(1); + assert_ne!( + bogus_version, real_version, + "fixture must construct a genuinely different version" + ); + + // Wrong version: must be detected, never silently mis-dequantized. + assert_eq!(floor.decode(10, bogus_version), None); + + // Matching version: must decode, to exactly what bucket_center gives. + let expected = floor.bucket_center(10); + assert_eq!(floor.decode(10, real_version), Some(expected)); + } + + // ── small direct-risk sanity checks on this file's own new code ──────── + + #[test] + fn calibrate_on_a_sample_with_no_finite_values_returns_none() { + let sample = [f64::NAN, f64::INFINITY, f64::NEG_INFINITY]; + assert!(calibrate(&sample).is_none()); + } + + #[test] + fn quantize_degenerate_bounds_returns_zero_matching_helix_guard() { + // An all-identical sample collapses the percentile window to + // lo == hi; `quantize` must not panic and must saturate every + // input to bucket 0 — the same degenerate-bounds guard + // `helix::quantize::RollingFloor::quantize` documents + // (`crates/helix/src/quantize.rs:100-102`). + let sample = vec![42.0_f64; 500]; + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + let (lo, hi) = floor.bounds(); + assert!( + hi <= lo, + "an all-identical sample should collapse the window, got [{lo}, {hi}]" + ); + assert_eq!(floor.quantize(0.0), 0); + assert_eq!(floor.quantize(42.0), 0); + assert_eq!(floor.quantize(1000.0), 0); + } +} diff --git a/crates/weather-poc/src/manifest.rs b/crates/weather-poc/src/manifest.rs index 1ed6fc19f..92db70cff 100644 --- a/crates/weather-poc/src/manifest.rs +++ b/crates/weather-poc/src/manifest.rs @@ -1 +1,615 @@ -//! Placeholder — owned by the manifest worker (plan §6.2). Filled in W1. +//! The ClassView-side field manifest: `(facet, pair, byte) <-> (ERA5 +//! variable, level, unit, floor id)`. +//! +//! # Why this exists (le-contract.md §2, slot purity) +//! +//! **"Labels and positions come from the ClassView, NEVER from a slot in the +//! payload."** The `NodeRow`'s value slab is dumb bytes; a byte carries no +//! variable name, no level number, no unit, no display label. The mapping +//! *"F1 pair 0 hi-byte = `v_component_of_wind` at 850 hPa, unit m/s, floor id +//! `v850`"* cannot live in the row — it lives here, as a committed data +//! artifact the bake reads. This module is ClassView-side metadata; nothing +//! it describes is ever written into a `NodeRow`. +//! +//! # The lane (plan `.claude/plans/weather-soa-bake-v1.md` §2.2) +//! +//! L4, `6 x (8:8)` palette256² (`.claude/v3/soa_layout/le-contract.md` §3): +//! each 16-byte facet is a 4-byte classid prefix followed by six `(lo, hi)` +//! byte pairs. `u8:u8` means two SEPARATE bytes — never widened to u16. Each +//! byte carries one ERA5 field quantised to 256 levels; a pair holds a +//! physically paired couple (e.g. wind components), so the pair's own +//! similarity is one 256×256 table read. +//! +//! # The committed W1 field set (plan §2.5) +//! +//! Three facets, F0 (surface), F1 (850 hPa), F2 (500 hPa); five occupied +//! pairs in F0 and three each in F1/F2. Every other pair is reserved-zero — +//! **absent from this manifest entirely**, per the "reserved slots are NOT +//! rows" rule: a slot with no entry is dormant by construction, expandable +//! later with zero layout change (RESERVE, DON'T RECLAIM). +//! +//! # `lo`/`hi` convention +//! +//! Within a pair written `A : B` in the plan's table, `A` is the LOW byte +//! and `B` is the HIGH byte — the pair's own left-to-right order, kept +//! consistent across the whole committed manifest (see the comment header +//! of `data/field_manifest_v1.tsv`). +//! +//! # Bar B0 — what is implemented here, and what is deferred +//! +//! Plan §4 W0's bar B0 is *"mutating one manifest entry must change the +//! bytes the bake writes for at least one cell."* **The bake does not exist +//! yet** (`D-WXS-4`, a separate, later deliverable), so that end-to-end half +//! cannot be exercised from this module. What IS implemented and tested +//! here is the manifest-level half of B0: resolution is real (a lookup +//! finds the right slot and nothing else), and a manifest whose two entries +//! collide on one `(facet, pair, byte)` slot is rejected outright. The +//! stay-silent twin — a manifest that differs byte-for-byte (reordered +//! rows) but is semantically identical still resolves every lookup +//! identically — is also tested. Wiring bar B0's end-to-end half is `D-WXS-4`'s +//! job, not this module's. + +use std::fmt; + +/// The exact header line every field-manifest TSV must start with (after any +/// leading `#` comment / blank lines, which are skipped). +const HEADER: &str = "facet\tpair\tbyte\tvariable\tlevel_hpa\tunit\tfloor_id"; + +/// The committed W1 field manifest, embedded at compile time so this crate +/// stays dependency-free (no TSV/CSV crate, no serde — plan §6.3). +const COMMITTED_MANIFEST_TSV: &str = include_str!("../data/field_manifest_v1.tsv"); + +/// Which byte of an `(8:8)` pair a manifest entry occupies. +/// +/// `u8:u8` means two separate bytes, never a widened `u16` — le-contract.md +/// §1/§3. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PairByte { + /// The low byte of the pair (the first-listed variable in the plan's + /// `A : B` table entries). + Lo, + /// The high byte of the pair (the second-listed variable). + Hi, +} + +impl fmt::Display for PairByte { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + PairByte::Lo => f.write_str("lo"), + PairByte::Hi => f.write_str("hi"), + } + } +} + +/// One row of the field manifest: the (facet, pair, byte) slot a single +/// ERA5 field occupies, plus its physical unit and calibration floor id. +/// +/// Nothing on this type is ever written into a `NodeRow` — it is +/// ClassView-side metadata the bake and any reader consults, per the +/// module-level slot-purity note. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ManifestEntry { + /// Facet index: `0` = F0 (surface), `1` = F1 (850 hPa), `2` = F2 + /// (500 hPa). Always `0..=2` for a validated manifest. + pub facet: u8, + /// Pair index within the facet, `0..=5`. + pub pair: u8, + /// Which byte of the pair this entry occupies. + pub byte: PairByte, + /// The ERA5 variable name, exactly as it appears in the Zarr store + /// (e.g. `u_component_of_wind`, `mean_sea_level_pressure`). + pub variable: String, + /// The pressure level in hPa, or `None` for a surface field (the TSV's + /// literal `surface` token). + pub level_hpa: Option, + /// The physical unit (e.g. `Pa`, `K`, `m s-1`, `kg kg-1`, `m2 s-2`, `1` + /// for a dimensionless fraction). + pub unit: String, + /// A stable identifier for the calibration floor this field uses. + /// Two entries sharing a `floor_id` share the same `[lo, hi]` + /// calibration (plan §2.3/D-WXS-3); none do in the committed W1 + /// manifest today. + pub floor_id: String, +} + +/// Everything that can go wrong parsing or validating field-manifest text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ManifestError { + /// The first non-comment, non-blank line did not match [`HEADER`] + /// exactly, or no such line existed at all (`line == 0`). + BadHeader { + /// 1-based source line number (`0` if no header line was found at + /// all). + line: usize, + /// The line actually found, if any. + found: String, + }, + /// A data row did not split into exactly 7 tab-separated columns. + WrongColumnCount { + /// 1-based source line number. + line: usize, + /// How many columns were actually found. + found: usize, + }, + /// The `facet` column was not `0`, `1`, or `2`. + FacetOutOfRange { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `pair` column was not in `0..=5`. + PairOutOfRange { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `byte` column was neither `lo` nor `hi`. + BadByteToken { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `level_hpa` column was neither the literal `surface` nor a + /// parseable non-negative integer. + BadLevel { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// Two entries claim the same `(facet, pair, byte)` slot — the manifest + /// is not load-bearing if this is possible (bar B0). + SlotCollision { + /// The colliding facet index. + facet: u8, + /// The colliding pair index. + pair: u8, + /// The colliding byte. + byte: PairByte, + /// The variable name of the first entry claiming the slot. + first_variable: String, + /// The variable name of the second entry claiming the slot. + second_variable: String, + }, +} + +impl fmt::Display for ManifestError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + ManifestError::BadHeader { line, found } => write!( + f, + "manifest line {line}: expected header {HEADER:?}, found {found:?}" + ), + ManifestError::WrongColumnCount { line, found } => write!( + f, + "manifest line {line}: expected 7 tab-separated columns, found {found}" + ), + ManifestError::FacetOutOfRange { line, value } => { + write!(f, "manifest line {line}: facet {value:?} is not 0, 1, or 2") + } + ManifestError::PairOutOfRange { line, value } => { + write!(f, "manifest line {line}: pair {value:?} is not in 0..=5") + } + ManifestError::BadByteToken { line, value } => write!( + f, + "manifest line {line}: byte token {value:?} is neither \"lo\" nor \"hi\"" + ), + ManifestError::BadLevel { line, value } => write!( + f, + "manifest line {line}: level_hpa {value:?} is neither \"surface\" nor an integer" + ), + ManifestError::SlotCollision { + facet, + pair, + byte, + first_variable, + second_variable, + } => write!( + f, + "manifest slot (facet={facet}, pair={pair}, byte={byte}) is claimed by both {first_variable:?} and {second_variable:?}" + ), + } + } +} + +impl std::error::Error for ManifestError {} + +/// A parsed, validated field manifest: an ordered list of [`ManifestEntry`] +/// rows plus the two resolution directions over them. +#[derive(Debug, Clone)] +pub struct FieldManifest { + entries: Vec, +} + +impl FieldManifest { + /// Parses and validates field-manifest text. + /// + /// Lines that are blank or start with `#` are skipped wherever they + /// occur. The first remaining line must equal [`HEADER`] exactly (a + /// permissive "starts with" check would be a silent-misread waiting + /// for a schema change). Every remaining line is a data row of exactly + /// 7 tab-separated columns. After every row parses, the full set is + /// validated for `(facet, pair, byte)` collisions. + /// + /// # Errors + /// + /// Returns the first [`ManifestError`] encountered — a malformed + /// header, a malformed row, or (only after every row has parsed + /// successfully) a slot collision between two otherwise-valid rows. + pub fn parse(text: &str) -> Result { + let mut header_seen = false; + let mut entries = Vec::new(); + + for (line_no, raw_line) in text.lines().enumerate().map(|(i, l)| (i + 1, l)) { + let line = raw_line.trim_end(); + if line.is_empty() || line.starts_with('#') { + continue; + } + + if !header_seen { + if line != HEADER { + return Err(ManifestError::BadHeader { + line: line_no, + found: line.to_string(), + }); + } + header_seen = true; + continue; + } + + let cols: Vec<&str> = line.split('\t').collect(); + if cols.len() != 7 { + return Err(ManifestError::WrongColumnCount { + line: line_no, + found: cols.len(), + }); + } + + let facet: u8 = cols[0] + .parse::() + .ok() + .filter(|v| *v <= 2) + .ok_or_else(|| ManifestError::FacetOutOfRange { + line: line_no, + value: cols[0].to_string(), + })?; + + let pair: u8 = cols[1] + .parse::() + .ok() + .filter(|v| *v <= 5) + .ok_or_else(|| ManifestError::PairOutOfRange { + line: line_no, + value: cols[1].to_string(), + })?; + + let byte = match cols[2] { + "lo" => PairByte::Lo, + "hi" => PairByte::Hi, + other => { + return Err(ManifestError::BadByteToken { + line: line_no, + value: other.to_string(), + }) + } + }; + + let variable = cols[3].to_string(); + + let level_hpa = if cols[4] == "surface" { + None + } else { + Some( + cols[4] + .parse::() + .map_err(|_| ManifestError::BadLevel { + line: line_no, + value: cols[4].to_string(), + })?, + ) + }; + + let unit = cols[5].to_string(); + let floor_id = cols[6].to_string(); + + entries.push(ManifestEntry { + facet, + pair, + byte, + variable, + level_hpa, + unit, + floor_id, + }); + } + + if !header_seen { + return Err(ManifestError::BadHeader { + line: 0, + found: String::new(), + }); + } + + validate_no_slot_collisions(&entries)?; + + Ok(FieldManifest { entries }) + } + + /// The committed W1 field manifest (`data/field_manifest_v1.tsv`). + /// + /// # Panics + /// + /// Panics if the committed artifact fails to parse or validate. That is + /// a build-time invariant, not a runtime one — the committed TSV is + /// meant to always be well-formed, and a session breaking it should see + /// the failure immediately rather than downstream. Use [`Self::parse`] + /// directly for a fallible read of arbitrary manifest text. + pub fn committed() -> Self { + Self::parse(COMMITTED_MANIFEST_TSV) + .expect("committed data/field_manifest_v1.tsv must parse and validate") + } + + /// Every entry in the manifest, in file order. + pub fn entries(&self) -> &[ManifestEntry] { + &self.entries + } + + /// Resolves `(variable, level_hpa)` to the `(facet, pair, byte)` slot it + /// occupies, or `None` if this manifest has no entry for it (e.g. a + /// pressure-level variable at a level outside the W1 set). + pub fn resolve(&self, variable: &str, level_hpa: Option) -> Option<(u8, u8, PairByte)> { + self.entries + .iter() + .find(|e| e.variable == variable && e.level_hpa == level_hpa) + .map(|e| (e.facet, e.pair, e.byte)) + } + + /// The inverse of [`Self::resolve`]: which entry (if any) occupies a + /// given `(facet, pair, byte)` slot. + pub fn resolve_slot(&self, facet: u8, pair: u8, byte: PairByte) -> Option<&ManifestEntry> { + self.entries + .iter() + .find(|e| e.facet == facet && e.pair == pair && e.byte == byte) + } +} + +/// Rejects a manifest where two entries claim the same `(facet, pair, +/// byte)` slot. `O(n^2)` over the entry count, which is fine at the W1 +/// scale (22 entries) and simple enough to audit by inspection. +fn validate_no_slot_collisions(entries: &[ManifestEntry]) -> Result<(), ManifestError> { + for i in 0..entries.len() { + for j in (i + 1)..entries.len() { + let a = &entries[i]; + let b = &entries[j]; + if a.facet == b.facet && a.pair == b.pair && a.byte == b.byte { + return Err(ManifestError::SlotCollision { + facet: a.facet, + pair: a.pair, + byte: a.byte, + first_variable: a.variable.clone(), + second_variable: b.variable.clone(), + }); + } + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Parses and validates: the committed TSV loads, every entry is in + /// range, and the per-facet slot count matches the plan §2.5 table + /// exactly (`==`, not `>=` — a permissive arity guard is a silent + /// misread waiting for a schema change). + #[test] + fn committed_manifest_parses_and_every_entry_is_in_range() { + let manifest = FieldManifest::committed(); + + assert_eq!(manifest.entries().len(), 22); + + for entry in manifest.entries() { + assert!(entry.facet <= 2, "facet {} out of range", entry.facet); + assert!(entry.pair <= 5, "pair {} out of range", entry.pair); + assert!(!entry.variable.is_empty()); + assert!(!entry.unit.is_empty()); + assert!(!entry.floor_id.is_empty()); + } + + let count = |facet: u8| { + manifest + .entries() + .iter() + .filter(|e| e.facet == facet) + .count() + }; + assert_eq!(count(0), 10, "F0 (surface): 5 occupied pairs x 2 bytes"); + assert_eq!(count(1), 6, "F1 (850 hPa): 3 occupied pairs x 2 bytes"); + assert_eq!(count(2), 6, "F2 (500 hPa): 3 occupied pairs x 2 bytes"); + } + + /// Can-it-fire: a manifest with two entries claiming the same + /// `(facet, pair, byte)` slot must be rejected. + #[test] + fn colliding_entries_are_rejected() { + let text = format!( + "{HEADER}\n0\t0\tlo\tmean_sea_level_pressure\tsurface\tPa\tmsl\n0\t0\tlo\tsurface_pressure\tsurface\tPa\tsp\n" + ); + + let err = FieldManifest::parse(&text).expect_err("colliding slots must be rejected"); + match err { + ManifestError::SlotCollision { + facet, pair, byte, .. + } => { + assert_eq!(facet, 0); + assert_eq!(pair, 0); + assert_eq!(byte, PairByte::Lo); + } + other => panic!("expected SlotCollision, got {other:?}"), + } + } + + /// Stay-silent twin of the collision guard: two entries on DIFFERENT + /// bytes of the same pair must be accepted, not flagged as a collision. + /// A validator that rejects everything carries exactly as much + /// information as one that accepts everything. + #[test] + fn non_colliding_entries_on_the_same_pair_are_accepted() { + let text = format!( + "{HEADER}\n0\t0\tlo\tmean_sea_level_pressure\tsurface\tPa\tmsl\n0\t0\thi\tsurface_pressure\tsurface\tPa\tsp\n" + ); + + let manifest = FieldManifest::parse(&text).expect("non-colliding rows must parse"); + assert_eq!(manifest.entries().len(), 2); + } + + /// Stay-silent twin (non-trivial, over the real committed data): a + /// manifest that is byte-different — the data rows in reverse order — + /// but semantically identical to the committed one must be accepted + /// AND must resolve every lookup identically. Mutating a reserved-zero + /// slot's description or reordering rows must change not one resolved + /// answer. + #[test] + fn reordering_the_committed_rows_changes_no_resolution() { + let lines: Vec<&str> = COMMITTED_MANIFEST_TSV.lines().collect(); + let header_idx = lines + .iter() + .position(|l| !l.trim().is_empty() && !l.starts_with('#')) + .expect("committed manifest must have a header line"); + let (head, data) = lines.split_at(header_idx + 1); + let mut reordered_data: Vec<&str> = data.to_vec(); + reordered_data.reverse(); + + let mut reordered_text = head.join("\n"); + reordered_text.push('\n'); + reordered_text.push_str(&reordered_data.join("\n")); + reordered_text.push('\n'); + + // Sanity: the text really is byte-different from the original. + assert_ne!(reordered_text, COMMITTED_MANIFEST_TSV); + + let original = FieldManifest::committed(); + let reordered = + FieldManifest::parse(&reordered_text).expect("reordered manifest must still validate"); + + assert_eq!(original.entries().len(), reordered.entries().len()); + for entry in original.entries() { + assert_eq!( + original.resolve(&entry.variable, entry.level_hpa), + reordered.resolve(&entry.variable, entry.level_hpa), + "resolution for {:?} at {:?} diverged after reordering", + entry.variable, + entry.level_hpa + ); + } + } + + /// Resolution is real: a known `(variable, level)` resolves to its + /// documented slot, and the inverse lookup round-trips back to the same + /// entry. + #[test] + fn resolves_a_known_slot_and_its_inverse_round_trips() { + let manifest = FieldManifest::committed(); + + let (facet, pair, byte) = manifest + .resolve("u_component_of_wind", Some(850)) + .expect("u_component_of_wind@850 must resolve"); + assert_eq!((facet, pair), (1, 0)); + assert_eq!(byte, PairByte::Lo); + + let entry = manifest + .resolve_slot(facet, pair, byte) + .expect("the inverse lookup must find the entry resolve() just returned"); + assert_eq!(entry.variable, "u_component_of_wind"); + assert_eq!(entry.level_hpa, Some(850)); + assert_eq!(entry.floor_id, "u850"); + } + + /// The other half of resolution being real: a `(variable, level)` pair + /// outside the W1 set resolves to `None`, not to a wrong slot. + #[test] + fn a_variable_level_outside_the_w1_set_does_not_resolve() { + let manifest = FieldManifest::committed(); + // temperature exists at 850/500 in W1, but not at 250. + assert_eq!(manifest.resolve("temperature", Some(250)), None); + // geopotential exists as a pressure-level field, never as a surface one. + assert_eq!(manifest.resolve("geopotential", None), None); + } + + /// Out-of-range rejection: `facet = 3` is rejected. + #[test] + fn facet_out_of_range_is_rejected() { + let text = format!("{HEADER}\n3\t0\tlo\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::FacetOutOfRange { .. }) + )); + } + + /// Out-of-range rejection: `pair = 6` is rejected. + #[test] + fn pair_out_of_range_is_rejected() { + let text = format!("{HEADER}\n0\t6\tlo\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::PairOutOfRange { .. }) + )); + } + + /// Out-of-range rejection: a bogus byte token (`mid`) is rejected. + #[test] + fn bogus_byte_token_is_rejected() { + let text = format!("{HEADER}\n0\t0\tmid\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::BadByteToken { .. }) + )); + } + + /// A malformed header is rejected rather than silently accepted with + /// misaligned columns. + #[test] + fn malformed_header_is_rejected() { + let text = "not\tthe\theader\n0\t0\tlo\tsome_var\tsurface\tPa\tid\n"; + assert!(matches!( + FieldManifest::parse(text), + Err(ManifestError::BadHeader { .. }) + )); + } + + /// A row with the wrong number of columns is rejected (`==`, not a + /// permissive minimum). + #[test] + fn wrong_column_count_is_rejected() { + let text = format!("{HEADER}\n0\t0\tlo\tsome_var\tsurface\tPa\n"); // 6 columns, not 7 + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::WrongColumnCount { line: 2, found: 6 }) + )); + } + + /// A `level_hpa` that is neither `surface` nor an integer is rejected. + #[test] + fn bad_level_token_is_rejected() { + let text = format!("{HEADER}\n0\t0\tlo\tsome_var\tnot_a_level\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::BadLevel { .. }) + )); + } + + /// `#`-prefixed comment lines and blank lines are skipped wherever they + /// occur, not just before the header. + #[test] + fn comments_and_blank_lines_are_skipped_anywhere() { + let text = format!( + "# leading comment\n\n{HEADER}\n# a mid-file comment\n0\t0\tlo\tsome_var\tsurface\tPa\tid\n\n# trailing comment\n" + ); + let manifest = FieldManifest::parse(&text).expect("comments must not break parsing"); + assert_eq!(manifest.entries().len(), 1); + } +}