diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 95aa64e42..5f1158146 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,55 @@ +## 2026-08-13 — weather-poc W0/W1 fleet (4 Sonnet workers, disjoint files; orchestrator-consolidated) + +4/4 completed, 0 errors. Plan `.claude/plans/weather-soa-bake-v1.md`; the first +grindwork wave on the crate `weather-substrate-poc-v2` named and never created. +Files were disjoint by construction (`key.rs` / `floor.rs` / +`manifest.rs`+`data/*.tsv` / `probes/weather-p1/era5_variable_census.*`); the +orchestrator owned `lib.rs`, both manifests, the root exclude entry and every +board file. Guardrails §1 pasted verbatim into all four briefs; no worker ran +cargo (rule 7) and all three Rust workers reported "not compiled, not run" +rather than claiming green. + +**D-WXS-1a census (Python, the one worker allowed to run).** Reads the store's +`.zmetadata`, emits the JSON, `--selftest` asserts all 10 constants the plan +carries. PASS, and re-run independently by the orchestrator. Guard +disable-verified twice — the worker broke a constant in a throwaway copy, and +the orchestrator repeated it against the committed file (exit 1, correct +"the fetched answer is authoritative" message). 0 arrays failed to classify. + +**D-WXS-2 key codec.** 5/5. Exhaustive 1,038,240-cell round-trip, collision- +checked. Bar B1 disable-verified by the ORCHESTRATOR, not asserted by the +worker: zeroing the HIP lat byte kills 3 tests (collision, ragged, +out-of-range); removing the seam split kills the wrap twin while the non-wrap +twin stays green — the pair discriminates in both directions. + +**D-WXS-3 floor.** 7/7. Bar B2 disable-verified: widening the "narrow" control +floor kills only the control, the stay-silent twin holds. The worker chose a +hand-rolled FNV-1a over `DefaultHasher` *because the latter is randomly seeded +per process* — a real trap, and one the suite does not cover, so the +orchestrator verified it separately: `floor_version` is byte-identical across +three separate process invocations. ε = 0.03 derived from the percentile +construction, not tuned post-hoc. + +**D-WXS-1 manifest.** 13/13, TSV 22 rows (F0 10 / F1 6 / F2 6, reserved slots +emit NO row — exactly plan §2.5). Collision guard disable-verified: removed → +only `colliding_entries_are_rejected` fails while BOTH stay-silent twins +(distinct lo/hi on one pair; row reordering) stay green. Bar B0's end-to-end +half is DEFERRED and said so in the module doc — the bake does not exist yet. + +**The wave's real find is a plan defect, not a code defect.** The key worker +noticed during implementation that §1.2 assigns one WHOLE byte per axis while +OGAR's cascade doctrine — the passage §1.1 leans on — specifies the axis bytes +nibble-interleaved (Morton). The plan deviated from the canon it cites and did +not say so. Recorded as plan §1.3a + board `D-WXS-2a` with a pre-registered +comparison, NOT silently rewritten: the prefix-scan claim holds under both +layouts, and rewriting a spec the moment a worker surfaces a consequence is how +a correction starts building. + +Gates (orchestrator, one shared `target/`): `cargo test --manifest-path +crates/weather-poc/Cargo.toml` **25/25**; `cargo clippy --all-targets -D +warnings` clean; `cargo fmt` clean. Commits `010da851` (scaffold), `a8a501bc` +(census + key), this one (floor + manifest). + ## 2026-08-11 — Workflow `wf_99d677e6-b45` — eval-plan verify/attack (13 agents; orchestrator-consolidated) 13/13 completed, 0 errors, 0 empty; ~4.42 M subagent tokens, 165 tool calls, diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index f63cb06bc..64213ed32 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,151 @@ +## 2026-08-12 — E-THE-REGIME-LADDER-MEASURED-RANGE-NOT-TURBULENCE-1 + +**Status:** FINDING `[G]` — measured, same run, found by an operator +question rather than by any gate. Qualifies +`E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND-REVERSED-1`, written one commit +earlier, whose *stronger* half it withdraws. + +**The operator asked: do we have a storm-modelling problem, since we do +not actively model vortices?** Checked, and the answer is yes, twice over. + +**First, the plain fact.** `substrate_comfort_d_cz_2_7.py` loads **only** +`mean_sea_level_pressure`. No wind, no vorticity, no rotation enters any +bar. D-CZ-0/1 fetched 10 m winds solely to *report* `spd_sigma`; nothing +consumed them. R4 "STORM" is a **scalar pressure-gradient** regime, and +the hypothesis it was built to test was about *"high velocity differences +/ turbulence"* — a dynamical property that appears nowhere in the +measurement. + +**Second, the confound that follows.** Chasing the question produced two +measurements that qualify the previous entry: + +| relationship | statistic | +|---|---| +| `L` vs the cell's `saturation` | Pearson **+0.917** (n=8) | +| `L̄[T]` vs the regime's own value range | **Spearman +1.000 — perfect** | + +`L` is essentially *how much of the target falls outside the donor's +codebook range*. A regime's own range determines how hard it is to cover +— R4's is ~**18×** R1's. So C3's monotone rise **restates the width +ordering**, which the `|∇p|` ladder itself produced (deeper low ⇒ steeper +gradient ⇒ wider range). Width is not the whole story — `R3 → R2` has a +*wider* donor yet 0.949 saturation, because the boxes sit at different +absolute pressure levels — so the real driver is **coverage** (width AND +offset), which `saturation` captures directly. + +C4 is inflated the same way: `Δ` was amended to RMSE **in Pa**, and RMSE +scales with range, so an absolute-Pa margin cannot be compared across +regimes differing 18×. Normalised as a ratio it reads **3.96 / 1.85 / +1.14 / 2.35** — not monotone, and R1 is the extreme, not R4. + +**What survives / what is withdrawn:** + +- **SURVIVES:** the hypothesis is **NOT SUPPORTED**. No sign flip in + either measure, normalised or not; `CAL-ABS` wins its own diagonal in + all four regimes. Unaffected by the confound. +- **WITHDRAWN:** *"cleanly reversed, monotonic"* and *"storms are LESS + forgiving of bad calibration."* As measured that says **wide-range + boxes are harder to cover with a foreign codebook** — arithmetic. + +**Why no gate caught this.** C1c *did* pass, and it was the right gate to +have — but it measured the structure of the **`|∇p|` field** (decay +length, Gini, tail ratio), not rotational structure, and the confound is +between the ladder's own discriminator and the codebook's coverage, not +between two regimes. A gate cannot catch a confound that lives *inside* +the variable it was told to trust. **The operator's question was the +instrument here; nothing in the apparatus was positioned to ask it.** + +**The transferable rule.** *When a regime axis is built from a scalar +derived from the same field the codec quantises, the axis and the codec's +difficulty are not independent.* Before reading a cross-regime encoding +result as physics, check whether the regime discriminator predicts the +encoding's own failure mode — here, `|∇p|` predicts range, and range +predicts saturation, and saturation IS `L`. A dynamical hypothesis needs +a **dynamical** discriminator (ζ = ∂v/∂x − ∂u/∂y, Okubo–Weiss) and a +**range-normalised** transfer metric, or coverage will masquerade as the +finding every time. + +**Filed as D-CZ-8**, the pre-condition for re-asking C3/C4 as a turbulence +question rather than a width question. + +## 2026-08-12 — E-THE-HYPOTHESIS-REFUTED-CLEANLY-AND-REVERSED-1 + +**Status:** FINDING `[G]` — measured, both pre-registered bars, one run. +`probes/weather-p1/substrate_comfort_d_cz_2_7.json`, plan §7. + +**The hypothesis this whole plan existed to test is refuted, and refuted +in the SHARPEST available form: not "no effect", but a clean monotonic +reversal.** The operator's hypothesis — *a badly-calibrated substrate +that maps dynamically preserves MORE structure in strong storms than a +well-calibrated absolute one* — was tested two ways, pre-registered, on +an equal-budget cross-swap over four regimes (calm → ocean → active → +storm): + +- **C3 (transfer loss)** required `L̄[R4] < L̄[R1]`. Measured: + `L̄` = **0.011 → 0.309 → 0.671 → 0.690**, R1→R4 — a **62× increase**, + monotonic across all four tiers. Storms are *less* forgiving of a + foreign absolute calibration, not more. +- **C4 (the crossover)** required a sign flip against the diagonal — + absolute winning in calm, dynamic winning in storm. Measured: absolute + wins its own diagonal in **all four** regimes, and its margin **grows** + from ~1 Pa (R1/R2) to **10.78 Pa** (R4) — the opposite of a crossover. + +Both measures point the same direction. This is not two noisy nulls; it +is one coherent, reversed relationship measured twice by independent +instruments (`ρ` on rank vs RMSE on the diagonal). + +**A real construction bug was found and fixed en route, via the gate +designed to catch exactly this class of defect.** C0 — *"both controls +must be WORSE than every real arm in every regime; if either matches a +real arm anywhere, that cell measures nothing"* — failed on its first run +in R1 and R4: `GEO-DEGENERATE` was built from `truth[:len//64]`, the +first slice of an array already shuffled by `rng.choice` for equal +budget. A random subsample of a flat array is not reliably narrower in +range than the whole array; that is a materially different construction +from D-CZ-1's correct one (`p[si,sj][:n_i,:n_i]`, a genuine 2-D corner). +Fixed by carrying each regime's full 2-D box alongside the flat +evaluation sample and building the degenerate donor from a real spatial +corner. **Disable-verified**: reverting to the flat-slice construction +reproduces the exact original failure; the fix reproduces the identical +real-arm numbers (C2–C6 unchanged) while making C0 pass cleanly in all +four regimes. The bug lived entirely in the control; nothing about the +real arms was ever wrong. Recorded because it is this arc's C0 gate doing +precisely the job it was built for — catching a defect in the apparatus +before it could contaminate a verdict — on the very first probe that +tried to use it for something other than a smoke test. + +**And it contradicts an earlier exploratory hint, on purpose, stated +plainly rather than averaged away.** §6.6 measured, *within R4 only*, a +single arm's saturation correlating with storm `\|∇p\|` at ρ = +0.444 +(p = 0.058, explicitly labelled not significant, not a result). That +correlation's *direction* matched the hypothesis. The properly-powered, +pre-registered, cross-regime test says the opposite. Where an +unpre-registered n=19 single-regime correlation and a pre-registered +cross-swap disagree, **the pre-registered result governs** — and this +entry exists so a later read of the plan does not quietly split the +difference between "p=0.058, direction matches" and "reversed, +monotonic, two independent measures" into a false middle. + +**C1c passed and licensed the whole exercise** — the regimes measurably +differ in correlation structure (decay length, Gini, tail ratio of +`|∇p|`), not merely in gradient magnitude, so the refutation above is a +finding about calibration under real structural variation, not an +artifact of four copies of one condition. + +**What is NOT concluded.** One box size (16°, 4225 cells), one variable +(MSLP), three timesteps plus 19 storms. C5 — the *other* half of "good +geometry vs badly calibrated" — was never run: the golden index floor +(N ≥ 2,550,409) is three orders of magnitude above what a box this size +can hold, so `GEO-GOLDEN-HI` has no admissible construction at this +scale. The geometry axis of the operator's original framing remains +completely untested. + +**Cross-ref:** `E-A-HORSE-RACE-IS-NOT-A-CROSS-SWAP-1` (the instrument +this run finally exercised); `E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1` +(the exact defect class C0 caught here); `E-A-FIGURE-CITED-TWICE-IS-NOT- +CONFIRMED-ONCE-1` (why §6.6's direction is not treated as confirmed by +this section merely because it was written down once already). + ## 2026-08-12 — E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1 **Status:** FINDING `[G]` — measured. Probe diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index de6ac45c8..60705ceb5 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,82 @@ +## 2026-08-13 — weather-soa-bake-v1 (PLAN; the missing Zarr→NodeRow bake) + +Plan: `.claude/plans/weather-soa-bake-v1.md`. **Does not supersede +`weather-substrate-poc-v2.md`** — §9 grades it part by part. The gap it closes: +poc-v2 names `crates/weather-poc` and declares "New repositories required: +ZERO", but the crate does not exist and the plan describes **no bake step at +all** — which is why the entire weather arc ran as Python probes over a Zarr +file, one variable, four hand-picked 16° boxes, three hand-typed timesteps. + +**The bake.** One cell = one node, one timestep = one Lance version. The key: +HEEL = `(lat>>6, lon>>6)` = the 16°×16° tile, HIP = position within it, TWIG +dormant-reserved — so a lat/lon grid is the literal-x/y binding OGAR's +256×256-centroid-tile cascade sanctions, and **the arc's hand-picked 16° boxes +become HEEL-prefix range scans**. The grid needs 5–6 of the 12 available +quaternary levels per axis (4⁵ ≥ 721, 4⁶ ≥ 1440), leaving room for 0.05° +without touching the stride. Three wrinkles stated, not designed around: both +axes tile **raggedly** (721 = 11×64+17; 1440/64 = 22.5), longitude is cyclic so +a seam-crossing box is **two** prefix ranges, and `MailboxId ≠ NiblePath` in +code so nothing leans on it. + +**The value.** L4 `6×(8:8)` palette256², one byte per ERA5 field, pairs being +physical couples (`(u:v)`, `(T:q)`) so a wind-vector distance is ONE 256×256 LUT +read. Argued against the whole le-contract §3 catalogue: L1–L3 are relational, +L5 kept only as the 3-D-wind alternative, L6 is the Odoo factoring, **L7 is +location and location is the KEY**, L8 is retired for this data (§12.12 RETIRED +the `Pair48` mint), G1–G3 are forbidden to new classes. **Stated deviation:** +L4's canonical Fisher-z read is replaced by the linear `helix::RollingFloor` — +licensed by le-contract §3's own monotone-bounded-continuous-field demarcation +and by §12.1's measurement (Fisher-z burns 228/256 buckets on ERA5 temperature). +Circular variables never ride an L1 byte (359°/1° at max L1 distance); wind +rides as `(u,v)`, and `wind_speed` is baked separately because §6.5 measured the +Jensen gap **is** gustiness (mean ratio 1.115, max gap 14.37 m/s). + +**Capacity, corrected.** "32 facets/cell" overstates the usable budget: 2 slots +are key+edges and `VALUE_TENANTS` carves the slab contiguously through offset +220 (committed assertion: **188 of 480 B**), so the free budget is **292 B = 18 +facets = 216 payload bytes** (discriminant 15 reserved for `BoardAggregates`; +weather takes 16, offset DERIVED). All 122 fields still fit at 1 B/field — +**one cell is one node** — but at 2 B/field they do NOT fit as 4+12 facets +(336 > 292). The 14 statics get their own classid and ONE version (baking them +per timestep would rewrite ~1.3 PB of constants). W1 bakes a minimal justified +set, not all 122, so the layout does not presuppose the fidelity gate's outcome. + +**Time.** `E-MARKOV-TEMPORAL-STREAM-1` applied literally: a timestep is a Lance +version, a time series is a version-range read (`QueryReference::at(v, rung)` + +`deinterlace`), the 19-storm set is a version set. Zero versioning code — the +`LanceCycleWriter` / `VersionedGraph` surface is consumed. Open risk, +pre-registered: 92,044 versions on one dataset is unmeasured (`D-WXS-6`, with a +KILL). + +**Re-homing.** `D-WXA-5` moves onto the substrate AND is re-specified: poc-v2's +ρ ≥ 0.98 is at serious risk of being vacuous because D-CZ-1 §6.4 measured +real-arm ρ spread at 3e-6…4.7e-5 — so the bar becomes (a) ρ ≥ 0.9996 (where a +real pair genuinely failed at 0.999556), (b) the shuffled control must fall +below 0.98, and (c) a 16/64/256-level ladder must be monotone BEFORE any verdict +(the can-it-DIFFER half). `D-CZ-8` moves on as ζ from a neighbour-key stencil — +with a falsifier the arc never had: **differencing amplifies quantisation +error**, so quantised-ζ vs true-ζ is reported per ζ-magnitude decile with a +laminar stay-silent twin. On §7.9: the full grid **does NOT dissolve the range +confound** (`L`~saturation +0.917, `L̄`~range +1.000 is arithmetic that 1.04 M +cells do not repeal); what it buys is (1) coverage-matched donor selection by +construction, (2) "donor" meaning a ζ-regime rather than a range once the shared +floor makes variables commensurable, (3) a regime sample that is not n = 4. + +**Splits.** Judgment (layout, key, every bar, the `D-WXA-5` re-spec, the +confound treatment, the mint) stays on Opus/main. Grindwork is +one-file-per-Sonnet-worker (`key.rs` / `floor.rs` / `lane.rs` / `bake.rs` / +`statics.rs` / `stencil.rs` / `metrics.rs`, one test file per bar), guardrails §1 +pasted verbatim, tag-files only. Orchestrator-only: `lib.rs` mod lines, the crate +manifest, `canonical_node.rs`, every board file. Root `Cargo.toml` untouched — +`weather-poc` is workspace-EXCLUDED. SIMD only via `ndarray::simd::*`; ndarray +optional, feature-gated, GIT-sourced (never an optional `path` dep). No +lance-family bump. Doc-only. + +**Also recorded:** `weather-substrate-poc-v2.md`'s `D-WXA-*`/`D-WXB-*`/`D-WXC-*` +ladder has **no rows on `STATUS_BOARD.md`** (`grep -c WXA` = 0, verified) — its +board hygiene was never discharged, which is a plausible mechanism for how a +whole arc ran past an unmet gate. + ## 2026-08-12 — substrate-comfort-zones-v1 REVISED (§2 rebuilt: horse race → cross-swap) Same file, `.claude/plans/substrate-comfort-zones-v1.md`, still **ACTIVE**, diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index d1b1fa7d4..2c7424e64 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -855,6 +855,7 @@ Membrane consumers can now pull BOTH halves of a render `classid` BBB-safely fro | PR | Merged | Title | What it added | |---|---|---|---| | *gap note* | — | **#781–#925 are NOT in this table** — carried by the dated sections above + `PR_ARC_INVENTORY.md`. Recorded 2026-08-12 (codex P2 on #930) rather than silently reconstructed; the table had stalled at #780. | — | +| **#946** | 2026-08-12 | D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix | MIXED (probe code + measurements + 2 epiphanies + a synthesis doc). **D-CZ-0 was marked DONE with no script and no JSON** — quoted in the plan, the arc, the LATEST_STATE row and 3 PR bodies; #945's self-audit called it "verified" while only comparing arc entry to PLAN (prose vs prose). Reproduced 4/9 rows (1.004/1.022/0.994/0.931); the 5 excluded land candidates are **unreproducible** (centres never recorded; no coordinates invented). `|∇p|` definition identified FROM DATA as **Pa/cell without cos(lat)** (max dev 0.069 vs next-best 0.398) → R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, **ORDER survives**, range 9.3× → **9.8×**. **D-CZ-1 PASS**: both controls lose on both metrics in all 4 regimes and in **19/19 storms**; `GEO-DEGENERATE` saturates 72–97 %; **C1b separation 6.28** vs ≥3. **The run AMENDED C4**: ρ saturated on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps ρ off-diagonal, **C4 moves to RMSE in Pa**; amendment carries its trigger and was propagated to header + bar. Committed **`--selftest`**, each assertion disable-verified (dropping tie-averaging **flips the sign** of heavy-ties ρ). Exploratory within-R4 ρ=+0.444, n=19, **p=0.0578 (not significant)** — and the tautology check does NOT dismiss it (confounds −0.035/+0.253/−0.081); committed WITH a not-a-result status string. **Plus `SUBSTRATE_FORMULA_MATRIX.md`**: 46 rated primitives (15 A / 4 B / 4 C / **20 D / 3 V** — half the inventory is negative), 14 known-vs-discovered pairs, 9 apparatus lessons, 13 gaps; built by re-extracting from committed artifacts (131 primitives, 2.35 M subagent tokens), **28 figures re-verified, 0 mismatches, 1 rounding fixed**. Every off-diagonal cross-swap cell remains unmeasured. | | **#944** | 2026-08-12 | `substrate-comfort-zones-v1` §2 rebuilt: horse race → CROSS-SWAP transfer matrix | MIXED (plan revision + epiphany + a review-caught correction). Operator ruling: the method is cross-swap under the premise the model CAPTURES the phenomenon but is NOT calibrated — so miscalibration is the measurement CONDITION, not an arm, and RMSE under a deliberately wrong calibration is bad BY DEFINITION. §2 is now a **4×4 donor × target matrix** `M[D][T]`; primary metric Spearman `ρ`; the hypothesis's quantity is the derived **transfer loss** `L[D][T] = ρ[T][T] − ρ[D][T]`. `CAL-ABS-OWN`/`CAL-ABS-FOREIGN` collapse into ONE arm's diagonal/off-diagonal. The dynamic arms have **no donor**, so `L ≡ 0` by construction — the property under test, and **C2 requires BOTH** that it hold exactly AND that `CAL-ABS` be proven non-degenerate through the same code path (the can-it-DIFFER gate W5 paid for). Two NEW bars for the operator's constancy-is-relative point: **C1b** MEASURES constancy (`separation ≥ 3`) instead of claiming it; **C1c** gives the suitability ASSUMPTION a falsifier — regimes must differ in autocorrelation decay + rank-distribution shape, not merely in `\|∇p\|`, and a null there VOIDS the reading. `L < 0` pre-registered as informative (a donor's wider range can legitimately beat an outlier-set diagonal). **CodeRabbit caught the sticky half**: §2.3 still named the FOREIGN comparison as the hypothesis while C4/board scored against the diagonal — two incompatible verdict criteria, the weaker holding the headline; fixed `1756c518`. Fifth instance this arc of a claim consistent with its own operands but inconsistent with a SIBLING claim. Unchanged: the 4-tier regime ladder + its three preflight corrections, equal budget, geometry axis, C0 gate, output contract. **D-CZ-1 (control losability) still Queued and still gates every later cell.** Zero code, zero probe runs, zero fetches. | | **#942** | 2026-08-12 | "mean" qualifier fix + `substrate-comfort-zones-v1` (calibration × regime, made falsifiable) | MIXED. Plan states the operator's hypothesis as an INTERACTION (a poorly-calibrated adaptive substrate does relatively better in strong/turbulent storms than in calm regimes) across **4** held-constant regimes × formula × calibration quality — the preflight SPLIT "flatland" into calm and active tiers: R1 CALM Amazon (`\|∇p\|` 10.2) → R2 OCEAN S-Pacific (14.9) → R3 ACTIVE W-Siberia (43.8) → R4 STORM (95.6), a 9.3× range. "Worse everywhere but least-worse in storms" ≠ "better only in storms", and the plan distinguishes them. D-CZ-0 preflight DONE (sample-composition arithmetic run before the first fetch — it is what forced the 4-way split; units annotated at coefficient definition). **D-CZ-1, the control-losability smoke test, is PRESCRIBED BY the plan but Queued — not yet run**, and gates every later cell. D-CZ-2..6 Queued. Also fixed CodeRabbit's Minor on #941 — #940's 25 %/9 % figures are MEANS over 19 storms, qualifier restored (4th instance this week of scope-lost-in-summary). | | **#940** | 2026-08-12 | W6 lands — vector-sum dipole model VOID by its own anti-vacuity control; stranded stratum empty by CT-F14's filter arithmetic; same-PR sign/units correction round | B0 VOID: single-geo R²=−0.104 (worse than the mean); both controls (permuted P_bow, P_bow rotated 90°) clear the `≤single-geo+0.03` ceiling. B3 stranded (`\|v_storm\|<8 m/s`): n=0 — `displacement_km≥250`/6h implies `\|v_storm\|≥11.57 m/s` for every admitted storm, `E-THE-DISPLACEMENT-FILTER-ATE-THE-STRANDED-STRATUM-1`. Same-PR: codex+CodeRabbit caught a sign-convention bug (`D=-spine(...)`, matching `low_pole_bearing()`'s own flip; verified offline that this leaves R²/B0/B1/B3 unchanged, confirmed on the actual re-run) and a units error (`c_bow` is km⁻¹ not dimensionless; replaced with the dimensionally valid `\|c_bow·P_bow\|` vs `\|D\|` metric — mean-over-19-storms: geo≈25%, bow≈9% of mean `\|D\|`). CT-F17's gate now moot for this model form. | diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 341c004ee..ae6035f2c 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,79 @@ +## 2026-08-12 — lance-graph #946 (MERGED) — D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix + +- **The finding that forced the run.** `substrate-comfort-zones-v1` §1's + nine-row regime preflight was marked **DONE** on `STATUS_BOARD`, quoted in + the arc entry, quoted again in the `LATEST_STATE` row, and its 9.3× headline + carried into three PR bodies — **with no script and no JSON producing any of + it.** Worse: #945's self-audit, run *specifically because* this arc keeps + shipping unbacked summary claims, reported those figures as "verified" while + only comparing the arc entry to the PLAN. Both are prose. **A figure cited in + two documents is cited twice, not confirmed once**, and an audit that compares + two documents measures their consistency — not the property anyone cares about. +- **Reproduced — 4 of 9 rows** (ratios 1.004 / 1.022 / 0.994 / 0.931). The five + EXCLUDED land candidates are **unreproducible**: their box centres were never + written down anywhere, and **no coordinates were invented** to fake them. +- **The `|∇p|` definition was never committed either** — so four candidates were + computed and the winner decided **from the data**: Pa per grid cell with **NO + cos(lat) metric** (max dev 0.069; next-best 0.398), a plain `np.gradient` over + the raw array. It understates the zonal gradient by `1/cos(lat)`, so **R3 at + 60 N is ~40 % low**. Metric-corrected the ladder is 10.3 / 15.5 / 61.2 / 100.9 + — **ORDER survives**, range widens 9.3× → **≈9.8×**. Regime axis stands; the + recorded magnitudes do not. +- **Two defects in the reproduction, both caught by its own guards.** Measuring + all 19 storms at the *preflight* timestep instead of each storm's own `t0` + (inverted R3/R4 and would have read as C1 failing); and a seam assertion firing + on a real storm at 353.4 E, where a plain longitude slice yields 58 columns + instead of 65. +- **D-CZ-1 PASSES.** Both controls lose to both real arms, on both metrics, in + all four regimes — and, after the sample-composition fix, in **19/19 storms**, + not merely at the median. `GEO-DEGENERATE` saturates **72–97 %**, so the + mechanism is visible rather than assumed. **C1b `separation` = 6.28** vs ≥ 3. +- **…and the run AMENDED C4.** `ρ` is **saturated on the diagonal** (real-arm + spread **3×10⁻⁶ … 4.7×10⁻⁵**) — C4 as pre-registered **could not have fired**. + `L` keeps `ρ` off-diagonal (four orders of range); **C4 moves to RMSE in Pa** + with `ρ` as a floor check. Legitimate because D-CZ-1's purpose is to test the + apparatus before the expensive cells and **no C4 cell has been scored**; + illegitimate the instant one has. Amendment carries its trigger and was + **propagated** to the header and the bar — the propagation failure a reviewer + caught on #944. +- **Committed `--selftest`, disable-verified.** ρ vs scipy incl. heavy ties, + exact ±1 anchors, `nan` on constant input; saturation 0.802 vs 0.008; + `rank_codec` monotone over 256 levels; seam box keeps 65 distinct columns. + Each assertion **broken on purpose** to confirm it can fail — dropping + tie-averaging **flips the sign** of the heavy-ties ρ (+0.054 vs −0.031), which + would have corrupted silently rather than crashing. +- **Exploratory, and it survives its own dismissal.** Within R4: + ρ(`GEO-DEGENERATE` ρ, storm `|∇p|`) = **+0.444**, n=19, permutation + **p = 0.0578** (200 000 perms) — **above 0.05, not significant**. The + tautology check does NOT hold (ρ vs box range −0.035; `|∇p|` vs box range + +0.253; ρ vs donor/box fraction −0.081), so neither confirmation nor easy + dismissal is available. Committed as `exploratory_within_r4` **with a status + string saying it is not a result**, so a later run cannot restate it as a + confirmation that was there all along. +- **Added: `probes/weather-p1/SUBSTRATE_FORMULA_MATRIX.md`** — the arc's rated + inventory. Built by re-extracting from the COMMITTED artifacts (7 parallel + readers + 1; **131 primitives, 2.35 M subagent tokens**), never from session + memory. **Two scales, deliberately unmerged:** fitness (A/B/C/D/V) × evidence + (`[G]`/`[H]`/`[S]`). **46 rated: 15 A, 4 B, 4 C, 20 D, 3 V — half the + inventory is a negative result.** A **C is not a bad grade** (Fisher-z alone + carries three verdicts: tail 8.3× win, level 4.7× loss, CI-frame not-a-win); + **D and V differ in kind** (D lost its test; V distinguished nothing). + 14 known-vs-discovered pairs, four recording the measurement going AGAINST the + prior. Figures verified against source JSONs: **28 checked, 0 mismatches, 1 + rounding fixed** (Rayleigh p 0.689 → 0.688). +- **Locked.** The regime ladder's ORDER; the C0 control gate as a real gate; + `ρ` for transfer loss; RMSE in Pa for the crossover. +- **Deferred.** D-CZ-2, 2b, 2c, 3..7 — **every off-diagonal cross-swap cell is + still unmeasured**. D-MTX-5 (refresh the matrix after they run). +- **Docs.** `EPIPHANIES.md` `E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1` + + `E-THE-METRIC-THAT-SEPARATES-ONE-COMPARISON-IS-BLIND-TO-ANOTHER-1`; + `STATUS_BOARD` D-CZ-0/D-CZ-1 updated, D-MTX-1..5 added; plan §6 RUN section. +- **Confidence.** `[G]` on D-CZ-1's gate, the reproduction, the definition + identification, and the ρ-saturation measurement — all committed and + re-runnable. `[H]` on the C4 amendment's *scope* (correct here, illegitimate + once a C4 cell exists). **Explicitly NOT a result:** the §6.5 RMSE-margin hint + and the §6.6 exploratory correlation, both labelled as such in the artifact. + ## 2026-08-12 — lance-graph #944 (MERGED) — `substrate-comfort-zones-v1` §2 rebuilt: horse race → CROSS-SWAP transfer matrix - **Added.** A rebuilt §2 for `.claude/plans/substrate-comfort-zones-v1.md` diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index e91a73b4b..c302ba1bb 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,47 @@ +## weather-soa-bake-v1 — Zarr → NodeRow, the missing bake (PRE-REGISTERED 2026-08-13) + +Plan: `.claude/plans/weather-soa-bake-v1.md`. The arc's Python-over-Zarr phase +ran because the Zarr→`NodeRow` path was never specified — +`weather-substrate-poc-v2.md` names `crates/weather-poc` and describes **no bake +step**. Waves W0–W5; every bar pre-registered and committed BEFORE its run, each +with a control that can lose and a stay-silent twin. + +| D-id | Deliverable | Wave | Status | Feeds | +|---|---|---|---|---| +| D-WXS-0 | classid mint for the weather-cell + statics classes (OGAR-side; `0x0F = Geo` exists, appid/classview open) | 0 | **Blocked (operator/OGAR)** | the bake must REFUSE to write under `0x0000_0000` | +| D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | **SHIPPED 2026-08-13** — `data/field_manifest_v1.tsv` (22 rows = F0 5 pairs + F1/F2 3 pairs each, reserved slots emit NO row) + `manifest.rs`, 13/13; collision guard **disable-verified** (removed → only `colliding_entries_are_rejected` fails, BOTH stay-silent twins stay green). Bar B0's end-to-end half (mutating an entry changes written bytes) DEFERRED — the bake does not exist yet | slot purity §2; bar B0 | +| D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | +| D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | +| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | +| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | +| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | +| D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | +| D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions | +| D-WXS-7 | **D-WXA-5 re-homed and RE-SPECIFIED** — ρ(code_dist, field_dist) via `jc::reliability::spearman` over whole-grid pairs. (a) ρ ≥ 0.9996 (the bar a real pair FAILED at 0.999556); (b) shuffled-codebook control < 0.98 (measured losable at 0.003–0.159); (c) 16/64/256-level ladder must be MONOTONE before any verdict | 3 | Queued | ⚠ poc-v2's ρ ≥ 0.98 is at risk of being vacuous — D-CZ-1 §6.4 measured real-arm ρ spread 3e-6…4.7e-5 | +| D-WXS-8 | cross-variable comparability at grid scale (≥4 variables, ≥2 units, ≥3 seasons); per-variable-floor control must LOSE | 3 | Queued | P2 (0.9997 vs 0.857–0.875) promoted from [H] or refuted | +| D-WXS-9 | ζ = ∂v/∂x − ∂u/∂y as a substrate read (neighbour-key stencil) + the differencing-amplifies-quantisation falsifier, reported per ζ-magnitude decile | 4 | Queued | bar B8; KILL ⇒ ζ becomes its own baked lane | +| D-WXS-10 | **D-CZ-8 re-homed** — ζ-percentile regime bands over the WHOLE grid + coverage-matched donors + range-normalised `L`; shuffled-ζ control; same-band stay-silent twin | 4 | Queued | the grid buys CONTROL of the §7.9 confound, NOT its dissolution; KILL if `L̄`~range ρ≈1.0 survives matching | +| D-WXS-11 | MEASURE a full-grid bake wall time; state the ~8.1 s serial extrapolation as the prior and confirm/correct it in the same artifact | 5 | Queued | replaces an extrapolation from ONE cycle (514 ms / 65,536 owners); D-KIA-A2 still unbuilt | +| D-WXS-12 | jc ↔ ndarray reliability agreement (poc-v2 D-WXB-4, carried verbatim; jc is the authority, degenerate input reported not folded) | 5 | Queued | every Phase-A/C number is computed with jc | + +**Capacity, corrected in the plan (§0.4).** "32 facets/cell" overstates the +usable budget: 2 slots are key+edges and `VALUE_TENANTS` carves the slab +contiguously — the committed assertion +(`canonical_node.rs` `value_tenants_contiguous_within_slab`) pins the current +Full carve at **188 of 480 B**, so the free budget is **292 B = 18 facets = +216 payload bytes**, not 384. All 122 fields still fit at 1 B/field — **one +cell is one node** — but at 2 B/field they do **not** fit as 4+12 facets. + +> **Board-hygiene note (2026-08-13):** `weather-substrate-poc-v2.md`'s +> `D-WXA-*` / `D-WXB-*` / `D-WXC-*` ladder has **no rows on this board** +> (`grep -c WXA` = 0, verified). None of those deliverables was ever built. +> The rows above do not supersede them; the poc-v2 ladder should be landed as +> its own block, marked NEVER BUILT, so the gap is visible rather than inferred. + +> **C5 note:** the global grid does **not** unblock `GEO-GOLDEN-HI`. The golden +> index floor needs N ≥ F(17)² = 2,550,409; the grid has 1,038,240. Short by +> ~2.5×, not by three orders of magnitude — still not constructible. + ## SUBSTRATE_FORMULA_MATRIX — the arc's rated inventory (2026-08-12) Document: `probes/weather-p1/SUBSTRATE_FORMULA_MATRIX.md`. Not a plan and not @@ -34,15 +78,16 @@ already run and it corrected two regime definitions before any bar existed. | D-id | Deliverable | Status | Feeds | |---|---|---|---| | D-CZ-0 | §1 regime preflight (`\|∇p\|` ladder, elevation-confound screen, speed-is-not-the-discriminator finding) | **DONE — and now REPRODUCED + partly corrected (§6.1/6.2).** It had NO committed script or JSON when marked DONE. 4 rows reproduce (1.004/1.022/0.994/0.931); the 5 EXCLUDED land candidates are **unreproducible** (centres never recorded). Definition identified: Pa per grid cell **without cos(lat)** — R3 ~40 % low; metric-corrected ladder 10.3/15.5/61.2/100.9, order survives, range 9.3× → **9.8×**. Original text:** — ladder R1 Amazon 10.2 → R2 ocean 14.9 → R3 W Siberia 43.8 → R4 storm 95.6 (9.3× range); 4 land candidates excluded on elev σ > 150 m | the regime axis all other rows score on | -| D-CZ-1 | C0 controls (shuffled codebook + degenerate geometry), losability-smoke-tested BEFORE the full run | **DONE — PASS** (`substrate_comfort_d_cz_0_1.py/.json`). Both controls lose to both real arms on BOTH metrics in ALL FOUR regimes; mechanism visible (`GEO-DEGENERATE` saturates 92–97 %). C1b `separation` = **6.28** vs the ≥ 3 bar. **AND the run amended C4**: `ρ` is SATURATED on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps `ρ` off-diagonal, C4 moves to RMSE in Pa (§6.4) | gates every cell — a control that can't lose voids its cell | -| D-CZ-2 | C1 regime-ladder stability across ≥3 timesteps | Queued | anti-cherry-pick on the whole regime axis | -| D-CZ-2b | **C1b constancy is measured** — `separation = (between-box range)/(mean within-box σ)` ≥ 3 | Queued | earns the phrase "held constant"; otherwise a caveat rides every cell | -| D-CZ-2c | **C1c the suitability ASSUMPTION** — regimes must differ in autocorrelation decay + rank-distribution shape, not only in `\|∇p\|` | Queued | a null here VOIDS the cross-swap reading (spread manufactured on the wrong axis) | -| D-CZ-3 | **C2 degenerate-row verification, both halves** — dynamic arms `L ≡ 0` exactly, AND `CAL-ABS` proven non-degenerate through the same code path | Queued | the can-it-DIFFER gate; without half (ii) a flat row proves nothing | -| D-CZ-4 | C3 **transfer loss** vs turbulence — `L̄[T] = mean_{D≠T} L[D][T]` strictly smaller in R4 than R1, reported with `occupancy`/`saturation` | Queued | "storms forgive bad calibration", with a mechanism attached | -| D-CZ-5 | **C4 the crossover** — `ρ(CAL-RANK) − ρ(CAL-ABS, D=T)` sign flip calm↔storm (vs the DIAGONAL, the hardest opponent) | Queued | **the operator's hypothesis, two-sided**; weak form vs `D≠T` reported separately | -| D-CZ-6 | C5 geometry floor on a SAMPLING-fidelity metric (NULL expected-plausible per W5 B4) | Queued | does the index floor bite where smoothing didn't | -| D-CZ-7 | C6 the transfer matrix (the deliverable) | Queued | comfort read off the diagonal, travel-cost off the off-diagonal | +| D-CZ-1 | C0 controls (shuffled codebook + degenerate geometry), losability-smoke-tested BEFORE the full run | **DONE — PASS** (`substrate_comfort_d_cz_0_1.py/.json`). Both controls lose to both real arms on BOTH metrics in ALL FOUR regimes; mechanism visible (`GEO-DEGENERATE` saturates **72–97 %** per regime — corrected 2026-08-12 from a stale 92–97 %, see plan §6.3; R4 is the low end at 0.7224). C1b `separation` = **6.28** vs the ≥ 3 bar. **AND the run amended C4**: `ρ` is SATURATED on the diagonal (real-arm spread 3e-6…4.7e-5) so C4 could not have fired — `L` keeps `ρ` off-diagonal, C4 moves to RMSE in Pa (§6.4) | gates every cell — a control that can't lose voids its cell | +| D-CZ-2 | C1 regime-ladder stability across ≥3 timesteps | **DONE — PASS.** `\|∇p\|` order R1 **2026-08-12 — rows re-cut, not restated.** The operator ruled that the > plan's §2 was built as a horse race where a cross-swap diagnostic belongs diff --git a/.claude/plans/substrate-comfort-zones-v1.md b/.claude/plans/substrate-comfort-zones-v1.md index 5b1b51202..56ba9c44a 100644 --- a/.claude/plans/substrate-comfort-zones-v1.md +++ b/.claude/plans/substrate-comfort-zones-v1.md @@ -1,11 +1,32 @@ # substrate-comfort-zones-v1 — where does each substrate formula feel at home? -> **Status:** ACTIVE, exploratory tier. Bars author-written and unaudited — -> fine for exploratory, said out loud (the -> `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` rule). -> No verdict-tier claim may be promoted out of this plan without an +> **Status:** RUN, §7 — the pre-registered hypothesis is **REFUTED** on this +> data. Bars author-written and unaudited — fine for exploratory, said out +> loud (the `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` +> rule). No verdict-tier claim may be promoted out of this plan without an > independent adversarial spec audit. > +> **⚠ HEADLINE, READ FIRST — see §7, and §7.9 for the confound.** +> **The hypothesis is NOT SUPPORTED:** there is no sign flip in either +> pre-registered measure, normalised or not — `CAL-ABS` wins its own +> diagonal in all four regimes. That much is solid. +> +> **But the stronger "cleanly reversed, monotonic" reading is WITHDRAWN** +> (§7.9, triggered by the operator asking whether storm modelling is sound +> given that vortices are not modelled at all). `C3`'s monotone `L̄` rise +> is **rank-correlated ρ = 1.000 with each regime's own value range**, and +> `L` tracks `saturation` at Pearson **+0.917** — so it largely restates +> *wide-range boxes are hard to cover with a foreign codebook*, which is +> arithmetic, not meteorology. `C4`'s "+10.78 Pa margin" is range-inflated +> for the same reason; normalised as a ratio it reads 3.96 / 1.85 / 1.14 / +> **2.35** — not monotone, and R1 is the extreme, not R4. +> +> **Scope, stated bluntly:** every bar here runs on a **scalar pressure +> field**. No wind, no vorticity, no rotation enters any metric. This plan +> answered *"absolute vs window-local encoding of a scalar field whose +> range varies by regime"* — a real question — but **not** the operator's +> original question about turbulence. +> > **Operator framing (2026-08-12), three messages:** > 1. *hold different situations constant — over water vs flatland vs storm > with high velocity differences / turbulence* @@ -487,10 +508,32 @@ regime axis stands; the recorded magnitudes do not. Both controls lose to both real arms, on both metrics, in **all four** regimes. The mechanism is visible rather than assumed: `GEO-DEGENERATE` -saturates **92–97 %** of the box because its donor patch's range is far too +saturates **72–97 %** of the box because its donor patch's range is far too narrow — which is exactly the failure mode the transfer matrix is built to measure. +> **⚠ FIGURE CORRECTED (2026-08-12, self-audit on #947).** This read +> **92–97 %** when first written, and that was TRUE of the numbers then on +> disk (R1 0.9496 / R2 0.9174 / R3 0.9718 / R4 0.9179). It went stale in +> the SAME PR: fixing R4 to measure each storm at its own `t0` moved R4's +> saturation to **0.7224**, and the range with it. Per-regime as committed: +> **R1 0.9496 · R2 0.9174 · R3 0.9718 · R4 0.7224**; across all 19 storms +> min **0.693** / median **0.843** / max **0.974**. +> +> **The verdict is unaffected** — 72 % is still overwhelming saturation and +> the gate passes 19/19 — but the figure was wrong in four files and two PR +> bodies. Seventh instance of the arc's recurring defect: a number true when +> written, stale once the artifact beneath it changed, and carried forward +> because the check compared prose to prose. Caught only by re-verifying +> against the JSON. +> +> **And the corrected number is more interesting than the wrong one.** The +> storm regime saturates *least* (0.72 vs 0.92–0.97 in the calmer tiers) — +> i.e. the degenerate donor hurts **less** where the field is strongest, +> which is independently the direction §6.6's exploratory correlation +> measured (ρ = +0.444). Two measurements that were never connected agree. +> Recorded as a coherence, NOT as evidence: §6.6 is still p = 0.0578. + | regime | CAL-ABS ρ | CAL-RANK ρ | SHUFFLE ρ | DEGENERATE ρ (satur.) | |---|---|---|---|---| | R1 | 0.999960 | 0.999993 | 0.058 | 0.380 (0.950) | @@ -602,3 +645,234 @@ control was run on this particular comparison; and the whole point of the cross-swap design is that a diagonal-vs-diagonal difference is not what the hypothesis is about. Recorded so a later run cannot present it as a confirmation that was there all along. + + +--- + +## §7 RUN — D-CZ-2..7, the cross-swap matrix (2026-08-12) + +Script `probes/weather-p1/substrate_comfort_d_cz_2_7.py`, results +`…_d_cz_2_7.json`, tag-file `exec-runs/…txt`. Run in the order the plan +requires: **C1c first**, because a null there would VOID the interpretation +before anything downstream is worth trusting. C0's construction had a real +bug, found and disable-verified fixed before any of C2–C6 was reported. + +### §7.0 A construction bug in C0 itself, caught by C0's own gate + +The first run of this probe **failed** C0 in two of four regimes: +`GEO-DEGENERATE` did not lose to the real arms in R1 or R4. Per §3's own +rule — *"if either matches a real arm anywhere, that cell measures +nothing and is reported VOID"* — nothing downstream could have been +trusted as written. + +**Cause:** the "degenerate donor" was built from `truth[:len//64]` — the +first slice of an array already subsampled by `rng.choice` for equal +budget. `rng.choice` returns no spatial order, so that slice is an +**ordinary random subsample**, not a narrow spatial patch — a materially +different construction from D-CZ-1's correct one +(`p[si,sj][:n_i,:n_i]`, a genuine 2-D corner). A random subsample of a +flat array is not reliably narrower in range than the whole array; by +chance it can nearly match it. + +**Fixed** by keeping each regime's full 2-D box alongside the equal-budget +flat evaluation sample, and building the degenerate donor from an actual +`n_i × n_i` corner of that box (`n_i = side // 8`), matching D-CZ-1 +exactly. **Disable-verified**: reverting to the flat-slice construction +reproduces the *exact* original failure (R1 and R4 fail, R2/R3 pass); +the fix reproduces the same real-arm numbers unchanged (C2–C6 identical +between the two runs) while making **C0 pass cleanly in all four +regimes**. The bug was isolated to the control; the real arms were never +wrong. + +### §7.1 C1c PASSES — the cross-swap interpretation is licensed + +| regime | decay length (cells) | Gini(\|∇p\|) | tail ratio (p99/p50) | +|---|---|---|---| +| R1 CALM | 25.5 | 0.4321 | **7.65** | +| R2 OCEAN | 32.0 | 0.2758 | 3.72 | +| R3 ACTIVE | 32.0 | 0.2262 | 1.94 | +| R4 STORM (median of 19) | 22.5 | 0.3062 | **2.95** | + +R4/R1 ratios: decay **0.88**, Gini **0.71**, tail ratio **0.385** — the +storm regime's gradient field is far *less* concentrated (a smaller +handful of cells carrying most of the gradient in calm regions; a more +uniformly elevated field in storms) than the calm regime's, a genuinely +different **shape**, not just a different mean. All three deviate from 1 +by ≥ 20 %. **The regimes differ in structure, not merely in `\|∇p\|`** — +the assumption behind the whole regime ladder holds. + +*(The 20 % threshold is this run's own operationalization of §3's +qualitative bar — the plan did not pre-register a number. Flagged as an +author choice; the raw ratios are reported above so a reader can apply a +stricter one and reach the same conclusion — the smallest deviation, +12 %, is on decay length, but tail ratio's 61 % deviation alone clears +any reasonable bar.)* + +**C1 (ladder) and C1b (separation) both PASS too**: the `\|∇p\|` order +R1 < R2 < R3 < R4 holds at all three tested timesteps, and separation is +**5.87–8.24** at every one — comfortably above the ≥ 3 bar. + +### §7.2 C2 PASSES both halves + +Dynamic arms (`CAL-RANK`, `CAL-FISHERZ`) show `L[D][T] = 0` **exactly** +for every donor in every regime — no leak. Through the *identical* code +path, `CAL-ABS`'s off-diagonal `L` ranges **0.011 (R1) → 0.947 (R4)** — +demonstrably non-degenerate. The can-it-DIFFER gate is satisfied. + +### §7.3 ⚠ C3 FAILS — and the failure is a clean, monotonic REVERSAL + +`L̄[T]`, the mean off-diagonal transfer loss for `CAL-ABS`: + +| R1 CALM | R2 OCEAN | R3 ACTIVE | R4 STORM | +|---|---|---|---| +| **0.011** | 0.309 | 0.671 | **0.690** | + +The bar required `L̄[R4] < L̄[R1]`. **Measured: `L̄[R4]` is 62× larger than +`L̄[R1]`, and the increase is monotonic across all four tiers.** This is +not "no shrinkage" — it is the pre-registered relationship holding in +**reverse**: transfer loss under a foreign absolute calibration grows, +not shrinks, as turbulence rises. Storms are **less** forgiving of bad +calibration on this data, not more. + +### §7.4 ⚠ C4 FAILS — same direction, same shape + +`Δ[T] = RMSE(CAL-RANK, own window) − RMSE(CAL-ABS, own diagonal)`, in Pa +(positive = absolute wins): + +| R1 CALM | R2 OCEAN | R3 ACTIVE | R4 STORM | +|---|---|---|---| +| +1.29 | +1.11 | +0.46 | **+10.78** | + +**No sign flip anywhere** — absolute wins its own diagonal in all four +regimes, and its margin of victory is **largest in the storm regime**, +not smallest. The strong form of the operator's hypothesis is refuted on +this data: a well-calibrated absolute encoding does not lose its edge as +turbulence rises here — it gains one. + +The **weak form** (`ρ(CAL-RANK) > ρ(CAL-ABS, D ≠ T)`) passes trivially +everywhere, exactly as pre-registered it would — dynamic beats a +genuinely *mis-calibrated* absolute by **329–2508 Pa** across regimes. +Per §3: *"nearly guaranteed by C2(i)… evidence of wiring, not of merit."* +Reported, not conflated with C4's real bar. + +### §7.5 C5 — NOT RUN, structural blocker + +`GEO-GOLDEN-HI` needs `N ≥ F(17)² = 2 550 409` before the golden lattice +behaves like the irrational it approximates (§1's own index-floor rule). +A 16° box at 0.25° resolution holds `65 × 65 = 4225` cells — **three +orders of magnitude below the floor**. `GEO-GOLDEN-HI` cannot be +constructed at box scale on this grid; the comparison has no admissible +high arm. Reported as a structural gap, not skipped silently, and not +faked with an interpolated sub-grid lattice (which would sample the +interpolator, not the field). + +### §7.6 C6 — the transfer matrix, delivered + +Full `donor × target` table for all five arms (`CAL-ABS`, `CAL-RANK`, +`CAL-FISHERZ`, `CAL-SHUFFLE`, `GEO-DEGENERATE`) × four regimes, every +`ρ` / `L` / `occupancy` / `saturation` / RMSE / bias cell raw, in +`substrate_comfort_d_cz_2_7.json` → `C6_matrix`. + +### §7.7 ⚠ This CONTRADICTS §6.6's exploratory hint — stated plainly + +§6.6 measured, *within* R4 only, ρ(`GEO-DEGENERATE` ρ, storm `\|∇p\|`) = ++0.444 (p = 0.058, not significant) — the *direction* of that correlation +matched the operator's hypothesis, and was recorded as a coherence with +the D-CZ-1 saturation numbers. + +**The properly-powered, cross-regime test says the opposite.** §6.6 was a +single-regime, single-arm, unpre-registered correlation at n=19. §7.3/§7.4 +are the pre-registered, cross-swap, equal-budget bars the whole plan was +built to produce. Where they disagree, the pre-registered cross-regime +result governs. §6.6 is retained as what it always was — exploratory, not +a result — and this section exists so a later read does not average the +two into a false middle. **The hypothesis is refuted on this data.** + +### §7.8 What this plan concludes + +The operator's hypothesis — *a badly-calibrated substrate that maps +DYNAMICALLY preserves MORE structure in strong storms than a +well-calibrated absolute one* — is **refuted, cleanly, on this box, this +variable, these three timesteps plus 19 storms**. Both pre-registered +measures of it (C3's transfer loss, C4's diagonal crossover) point the +same direction, monotonically, not merely "no effect." + +**What is NOT concluded:** that miscalibration is never more forgivable +under turbulence, anywhere, on any variable, at any scale — this is one +box size, one variable (MSLP), one gradient definition, and C5's absence +means the geometry axis (the *other* half of "good geometry vs badly +calibrated") was never tested at all. The regime axis and the C0/C1/C1b/ +C1c apparatus all hold; the specific calibration hypothesis they were +built to test does not. + + +### §7.9 ⚠ CONFOUND FOUND POST-RUN — C3's monotonicity is largely RANGE, not turbulence + +**Trigger (operator, 2026-08-12):** *do we have a problem with the storm +modelling, since we do not actively model vortices?* The question is +correct, and chasing it surfaced a confound that qualifies §7.3 and §7.4. + +**Fact first:** `substrate_comfort_d_cz_2_7.py` loads **only** +`mean_sea_level_pressure`. No wind, no vorticity, no rotation. D-CZ-0/1 +loaded 10 m winds solely to *report* `spd_sigma`; no bar ever consumed +them. R4 "STORM" is therefore characterised by a **scalar pressure-gradient +magnitude**, not by the rotating structure that makes a storm a storm. + +**The confound that follows, measured:** + +| relationship | statistic | +|---|---| +| `L` vs the cell's `saturation` | Pearson **+0.917**, Spearman **+0.833** (n=8) | +| `L̄[T]` vs the regime's OWN value range | **Spearman +1.000 — perfectly monotone** | + +`L` is essentially a function of **how much of the target's distribution +falls outside the donor's codebook range**. A regime's own range is what +makes it hard to cover: R4's implied range is ~**18×** R1's (≈7075 Pa vs +≈386 Pa). So `L̄` rising monotonically R1→R4 substantially **restates the +regimes' width ordering**, which the `|∇p|` ladder itself produced (a +deeper low means a steeper gradient *and* a wider box range). + +Width alone is not the whole mechanism — `R3 → R2` has a *wider* donor +(log₂ ratio +1.32) yet saturation 0.949, because the two boxes sit at +different absolute pressure levels. The true driver is **coverage** +(width **and** offset), which `saturation` captures directly and which the ++0.917 correlation measures. + +**C4 is range-inflated the same way.** `Δ` was amended to RMSE **in Pa** +(§6.4), and RMSE scales with the field's range, so an absolute-Pa margin is +not comparable across regimes differing 18× in range. Normalised as a +ratio, C4 reads **3.96 / 1.85 / 1.14 / 2.35** (R1→R4) — **not monotone**, +and R1, not R4, is where absolute wins by the largest factor. + +**What survives, and what does not:** + +- **SURVIVES — the hypothesis is NOT SUPPORTED.** There is no sign flip in + either metric, normalised or not; `CAL-ABS` wins its own diagonal in all + four regimes. That conclusion is unaffected by the confound. +- **DOES NOT SURVIVE — "cleanly reversed, monotonic."** §7.3's monotone + `L̄` increase is confounded with range (ρ = 1.000), and §7.4's "margin + grows to 10.78 Pa" is range-inflated. **"Storms are LESS forgiving of bad + calibration" is withdrawn as a physical claim** — as measured it says + *wide-range boxes are harder to cover with a foreign codebook*, which is + arithmetic. +- **C1c still passes** and still licenses the design — but note what it + measured: structure of the **`|∇p|` field**, not rotational structure. + It cannot have caught this, because the confound is between the + ladder's own discriminator and the codebook's coverage, not between two + regimes. + +**The deeper gap this exposes.** The hypothesis was about *"high velocity +differences / turbulence"* — a **dynamical** property. Every bar in this +plan is computed on a **scalar pressure field**. A vortex is not +represented anywhere: not in the regime definition, not in the codebook, +not in any metric. So this plan has tested *"badly-calibrated absolute vs +window-local encoding of a scalar field whose range varies by regime"* — +which is a real question, and answered — but it is **not** the operator's +original question about turbulence, and should stop being described as if +it were. + +**Consequence for the next run:** a genuine turbulence regime needs a +**rotational** discriminator (relative vorticity ζ = ∂v/∂x − ∂u/∂y, or +Okubo–Weiss), and a range-matched or range-normalised transfer metric so +coverage cannot masquerade as the finding. Both are pre-conditions for +re-asking C3/C4 as a turbulence question rather than a width question. diff --git a/.claude/plans/weather-soa-bake-v1.md b/.claude/plans/weather-soa-bake-v1.md new file mode 100644 index 000000000..89b2f24ff --- /dev/null +++ b/.claude/plans/weather-soa-bake-v1.md @@ -0,0 +1,803 @@ +# weather-soa-bake-v1 — Zarr → NodeRow, the missing bake, and the re-homing of the open weather work onto the substrate + +> **Status:** PLAN (2026-08-13). Doc-only. No code shipped, no board file written by this +> plan (the orchestrator is the sole writer of `.claude/board/*`; proposed rows are in §11). +> **Does not supersede** `weather-substrate-poc-v2.md` — §9 states part-by-part what survives, +> what is superseded by having the data on the substrate, and what was never built. +> +> **The one-sentence reason this plan exists:** an extended weather R&D arc ran entirely as +> Python over a Zarr file because **the Zarr→`NodeRow` path does not exist and no plan ever +> specified one** — `weather-substrate-poc-v2.md` names `crates/weather-poc`, declares "New +> repositories required: ZERO", and then describes encoder arms with **no bake step at all**. + +--- + +## §0 Corrections to the framing this plan was commissioned under + +Stated first, per the workspace rule that a plan which designs around a wrong premise is worse +than one that names it. + +**§0.1 — `D-WXA-5` is NOT on `STATUS_BOARD.md`. Neither is any `D-WXA-*` / `D-WXB-*` / +`D-WXC-*` row.** `grep -c "WXA" .claude/board/STATUS_BOARD.md` = **0** (verified this session, +and re-verified by the orchestrator before landing). The D-ids exist only inside +`.claude/plans/weather-substrate-poc-v2.md` §2–§4 and in the `INTEGRATION_PLANS.md` prose entry +dated 2026-08-10. The board-hygiene rule (`CLAUDE.md` § Mandatory Board-Hygiene Rule, row "A +new D-id / deliverable") was **not discharged** for that plan. Consequence: the weather POC's +deliverable ladder has never been visible on the deliverable dashboard, which is a plausible +mechanism for how a whole arc ran past it in Python without anyone tripping over an unmet gate. + +**§0.2 — the label "the C2 gate" does not match the artifacts.** In +`weather-substrate-poc-v2.md` §2, `D-WXA-5` is *"the gate"* (PASS: ρ ≥ 0.98 for ≥1 arm **AND** +the shuffled-codebook control FAILS). `C2` is a bar in a *different* plan +(`substrate-comfort-zones-v1.md` §3 — the degenerate-row verification). They are unrelated +bars. This plan uses `D-WXA-5` and never the C2 label. + +**§0.3 — `D-WXA-5` as pre-registered is at serious risk of being VACUOUS, and that is the +single most load-bearing thing this plan has to say about it.** `substrate-comfort-zones-v1.md` +§6.4 **measured** that Spearman ρ *saturates* on smooth pressure fields at 256 levels: the +spread between the two real arms was **3×10⁻⁶ … 4.7×10⁻⁵**, and the plan's own C4 bar +*"could not have fired."* `D-WXA-5`'s bar is ρ ≥ 0.98 between code-distance and +field-distance — four orders of magnitude below where real arms actually land. **Every real +arm will pass it.** A bake-off in which every arm passes has measured nothing, which is what +`D-WXA-5`'s own anti-vacuity paragraph says. §5 below re-specifies the bar. This is not a +licence to weaken the gate; it is the `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1` rule applied +to the gate itself, using a number already committed to an artifact. + +**§0.4 — "32 × (4+12) facets per cell" and "33.2 M facets per timestep" are arithmetically +right and semantically misleading.** `GUIDS_PER_NODE = 32` is const-asserted +(`canonical_node.rs:805-808`, `512 / 16`), and 1,038,240 × 32 = 33,223,680. But: **2 slots are +`key` and `edges`**, and the 480-byte value slab is **not free space** — `VALUE_TENANTS` carves +it *contiguously, gap-free, compile-asserted*. The committed assertion is exact +(`canonical_node.rs:2292-2307`, `value_tenants_contiguous_within_slab`): **the current Full +carve uses 188 of 480 B** (kanban×Rubicon 8 + autopoiesis triangle 3×12 = 36 + TEKAMOLO facet +16 + CausalWitness facet 16), ending at row offset **220**. Discriminant **15 is reserved for +`BoardAggregates`** (`canonical_node.rs:1027-1033`); a new tenant takes 16. + +**Free budget for a weather lane today = 480 − 188 = 512 − 220 = 292 bytes** = 18 whole +16-byte facets (+4 spare bytes) = **216 bytes of facet payload**. §2 does the capacity +arithmetic against that number, not against 384. + +**§0.5 — the ~8.1 s/timestep figure is a serial extrapolation from ONE measured cycle, and it +is also a category error.** The measurement (`AGENT_LOG.md` 2026-08-05, PROBE-IGNITION-64K) +is: **65,536 real `MailboxSoA<4>` owners**, build 8.4 s, c1 cast 225 ms, **seal+apply 514 ms**, +c2 rest 73 ms. 1,038,240 / 65,536 = 15.84 ⇒ 16 × 514 ms = **8.22 s** — arithmetic `[G]`, +*conclusion* **CONJECTURE**, for three independent reasons: + +1. It assumes seal+apply is **linear in owner count**. `D-KIA-A2` — the pre-registered + parallelism falsifier (median-of-5, ≥2× at ≥4k owners, ±10 % stay-silent twin) — is + **`Queued` and unbuilt**. Nothing has been measured about scaling. +2. The measured run bought scale on the **owners** axis only (`MailboxSoA<4>`, **one populated + row per owner**). Weather's shape is the *rows-per-owner* axis, which that probe explicitly + did not exercise. +3. **A bake is not a mailbox cycle.** Writing 1,038,240 rows into a Lance version is a + columnar write; seal+apply is the cognition loop *over* rows that already exist. Quoting + 514 ms/cycle as a bake cost conflates two different operations. §4's `D-WXS-11` replaces + the extrapolation with a measurement rather than refining it. + +Everywhere below, the 8.1 s figure appears only as **CONJECTURE (serial extrapolation from one +measured cycle; `D-KIA-A2` unbuilt)**. + +**§0.6 — variable-census inputs used by this plan** (from a live `.zmetadata` read of +`1959-2022-6h-1440x721.zarr`, and **required to be committed as a re-runnable probe** — +`D-WXS-1a` — because a census asserted in prose is exactly the chat-only-figure defect this +arc has already catalogued three times): 17 time-varying surface + 7 pressure-level × 13 +levels = 91 + 14 static = **122 fields/cell**; time axis 92,044 six-hourly steps. Per +timestep: 1,038,240 × 122 = **126,665,280 values** landing in 1,038,240 × 512 B = +**531,578,880 B = 0.495 GiB** (arithmetic `[G]`). + +--- + +## §1 The key — does HEEL/HIP/TWIG address 721 × 1440 directly? + +**Verdict: YES, directly, with three named wrinkles. This is the literal-x/y case OGAR's +cascade doctrine explicitly sanctions, and it is the cleanest instance of prefix-routing in +the workspace.** + +### §1.1 The arithmetic + +`/home/user/OGAR/CLAUDE.md` § *Tier interpretation — 256×256 CENTROID TILE*: each tier's 16 +bits reads as a **256×256 tile** = two axes × one byte, nibble-interleaved; and the rigour +condition is **256 = 4⁴ — each codebook is a 4-level 4-ary centroid hierarchy**, so a byte's +nibbles are ancestry and `is_ancestor_of` = containment. Three tiers × 4 quaternary levels = +**12 quaternary levels per axis** = 4¹² = 16,777,216 addressable positions per axis. + +The grid needs 721 lats and 1440 lons. 4⁵ = 1024 ≥ 721; 4⁶ = 4096 ≥ 1440. **The 0.25° global +grid consumes 5–6 of the 12 available quaternary levels per axis** — leaving room for a 0.05° +grid (3600 × 7200) inside the same three tiers without touching the stride. + +**And the hierarchy condition holds *exactly*, not approximately.** OGAR's warning is +*"flat k-means-256 breaks this; hierarchical 4⁴ preserves it."* For a literal x/y grid the +"centroid codebook" is the **identity mipmap**: a coarse tile *contains* its fine cells by +construction. There is no codebook to train and nothing to certify — `is_ancestor_of` is exact +containment. That makes the weather grid the **strongest available test case** for the whole +prefix-routing claim, and §4's `D-WXS-2` turns it into a test rather than a remark. + +### §1.2 The proposed assignment + +| field | content | width used | +|---|---|---| +| `classid` (bytes 0..4) | the weather-cell class (§1.4 — **not minted by this plan**) | u32 | +| `HEEL` (4..6) | `(lat_idx >> 6, lon_idx >> 6)` — the **16° × 16° tile**: lat 0..11, lon 0..22 | 2 bytes, one per axis | +| `HIP` (6..8) | `(lat_idx & 63, lon_idx & 63)` — position within the tile | 2 bytes, 6 bits each | +| `TWIG` (8..10) | **0 — dormant, reserved** (zero-fallback: "not consulted", never "compacted") | — | +| bytes 10..16 | **untouched** — V1-legacy u24 tail; new units mint the V3 facet, never `NodeGuid::new`. The bake calls `mint_for(classid_read_mode(c).tail_variant, …)`. | — | + +**The payoff, and it is the headline of this section:** the four hand-picked **16° boxes** the +entire Python arc ran on **are HEEL values**. `substrate_comfort_d_cz_2_7.py`'s `box_index()` +becomes a **key predicate**, and a box read becomes a **HEEL-prefix range scan** — the +"prerender from keys alone, zero value decode" property applied to a physical field. The +65 × 65 = 4225-cell box of the Python is the 64 × 64 HIP space of one HEEL tile (plus the +inclusive edge row/column the probe took). + +`TWIG` stays dormant deliberately: it is the reserved home for a **pressure level** if a later +wave decides a (cell, level) is its own node, or for a sub-0.25° refinement. RESERVE, DON'T +RECLAIM — a non-zero mint later wakes it with **zero `ENVELOPE_LAYOUT_VERSION` change**. + +### §1.3 The three wrinkles — stated, not designed around + +1. **Neither axis tiles evenly at a power-of-two tile size.** 721 = 11 × 64 + 17 (the grid is + pole-inclusive: 720 + 1), and 1440 / 64 = **22.5**. So the last tile on each axis is + **ragged** (17 rows; 32 columns). Decision: **accept ragged tiles**, address by index with + half-open tile ranges, and *never* pad — padding would invent cells that do not exist and + would silently enter any global statistic. `D-WXS-2`'s bar asserts the ragged tiles + round-trip exactly. +2. **Longitude is cyclic; Morton prefix ranges are not.** A box crossing the 0°/360° seam is + **two prefix ranges, not one**. The Python already does `% len(LONS)`; the Rust key API must + return a **range-set**, and the type must make a single-range answer impossible to write by + accident. Latitude has no wrap (it has poles — a different, non-cyclic degeneracy: the pole + *row* is a single physical point repeated 1440 times, which will show as an exact-tie + plateau in any rank statistic; the Python's tie-averaging `spearman` already handles it, + and the Rust side must too). +3. **`MailboxId ≠ NiblePath` in code** (`le-contract.md` §5, discrepancy 3: `MailboxId = u32`, + `NiblePath{path: u64, depth: u8}`, no conversion, no shared trait). The three-tier doc's + *"MailboxId IS the NiblePath"* is doc-only and awaiting a ruling. **This plan must not lean + on it.** A weather cell's key is a `NodeGuid`; whether a HEEL tile is also a mailbox id is + out of scope and is not assumed anywhere below. + +### §1.3a ⚠ UNDECLARED DEVIATION from the OGAR cascade doctrine — whole-byte axes, not nibble-interleaved (found 2026-08-13 by the D-WXS-2 worker, during implementation) + +§1.2 above assigns **one whole byte per axis** (`HEEL` = byte 4 lat, byte 5 lon; +`HIP` = byte 6 lat, byte 7 lon). OGAR's cascade doctrine — the same passage §1.1 +leans on for the 256×256 tile reading — specifies the two axis bytes +**nibble-interleaved**, and names the consequence explicitly: *"the x/y +nibble-interleave = alternating-axis refinement (Morton in centroid space)."* +§1.2 therefore **deviates from the canon it cites, and did not say so.** Recorded +here rather than silently corrected. + +**What survives unchanged.** The plan's actual load-bearing claim — +*a 16° box becomes a HEEL-prefix range scan* — holds under **both** layouts: +fixing both HEEL bytes fixes a prefix either way, so a tile-aligned box is one +contiguous byte range regardless. The `is_ancestor_of` = containment argument in +§1.1 also survives: interleaving governs how two axes **share** a tier's 16 bits, +not the per-tier ancestry the identity mipmap supplies. + +**What actually differs.** Byte-lexicographic order over whole-byte axes is +**row-major** (lat-major, lon-minor), not locality-preserving. Consequence for +`D-WXS-9` (the ζ stencil): a `lon ± 1` neighbour is adjacent in key order, while +a `lat ± 1` neighbour is 1440 cells away. Under Morton both are near. An +arbitrary (non-tile-aligned) box is also not one contiguous range under either +layout, but Morton bounds it in far fewer ranges. + +**Not changed now, deliberately.** Three reasons: the prefix-scan claim holds as +written; which layout is better *for this workload* is a measurable question and +this arc's discipline is measure-don't-assume; and rewriting a spec the moment a +worker surfaces a consequence is how a correction starts building +(`E-ACTOR-IS-NOT-THE-PHASE-PATH-1`'s recorded lesson: *when correcting an +over-built design, the correction must not itself build*). The shipped +`key.rs` implements §1.2 as specified and its five falsifiers are +disable-verified green. + +**The decision this owes.** A pre-registered comparison — row-major vs Morton +over the ζ-stencil neighbour read (`D-WXS-9`) and over a non-tile-aligned box +scan — with the metric stated before the run. Until it runs, §1.2's layout is +**a stated deviation, not a ruling**, and anything downstream that assumes Morton +locality is unfounded. Tracked as `D-WXS-2a` on `STATUS_BOARD.md`. + +**A second, smaller item from the same worker, resolved by convention and not by +test:** `box_ranges` treats `lon_lo == lon_hi` as *wrap the whole circle*, not as +*empty box*. Neither reading is forced by the spec. It needs a test pinning the +chosen one, or an API that makes the ambiguity unrepresentable. + +### §1.4 classid — a mint decision, NOT taken here + +`0x0F = Geo` already exists in the OGAR domain table; free domains are `0x03–0x06` +(`weather-substrate-poc-v1` §, carried into v2 §5). The composed classid is canon-high: +`domain ++ appid` in the high u16, **ClassView selector** in the low u16. Minting is an +OGAR-side, operator-gated act — **this plan does not mint it and does not invent a placeholder +that could reach a committed dataset.** `D-WXS-0` (§4, W0) is the blocked row. Until it is +resolved the bake **must refuse to write** rather than write under `0x0000_0000` (the +zero-fallback ladder owns that value and a weather dataset carrying it is unroutable garbage +with no way to tell it apart from a bootstrap row). + +--- + +## §2 The value — which variables ride which facets, in which layout + +### §2.1 Capacity, against the real budget + +Free slab after existing tenants: **292 bytes** (§0.4). As 16-byte facets: **18 facets = +216 payload bytes**. + +| scheme | payload bytes needed | fits as 4+12 facets? | fits as a raw byte lane? | +|---|---|---|---| +| all 122 fields @ 1 B | 122 | **yes** (11 facets = 176 B of slab) | yes | +| 108 time-varying fields @ 1 B (statics split out, §2.4) | 108 | **yes** (9 facets = 144 B) | yes | +| all 122 fields @ 2 B | 244 | **NO** (21 facets = 336 B > 292) | yes (244 ≤ 292) | + +**The headline survives, with the budget corrected: the entire ERA5 state for a cell — all 17 +surface, all 91 upper-air across every one of the 13 pressure levels, all 14 statics — fits +inside one canonical 512-byte `NodeRow` at 1 byte/field, with room to spare. One cell is ONE +node.** No spillover table, no second row, no level-as-node scheme. + +But the 2-byte column matters and must not be skipped: **at 2 bytes/field the state does *not* +fit as proper 4+12 facets** — only as an unprefixed raw byte lane (the shape `FrozenStyle` / +`LearnedStyle` / `ExploreStyle` already use: 12 raw palette bytes, no classid prefix). If +`D-WXS-7` (§5) shows 1 byte/field is insufficient for some variable, the fallback is +**per-variable 2-byte stacked floors on a subset**, not a global 2-byte scheme. + +### §2.2 The layout choice, argued against the §3 catalogue + +Rejected, with reasons: + +| # | layout | why not | +|---|---|---| +| L1/L2/L3 | rails `part_of:is_a` / `memberof:members` / mereology:taxonomy | Relational planes. A cell's temperature is not a relation to anything. The classview is a *focus lens*: relational data focuses rails, and this data is not relational. | +| L5 | `4×(8:8:8)` SPO triplets | The SPO reading is wrong for a physical field. **Kept as the named alternative** for a 3-D wind lane `(u,v,w)` per level if `vertical_velocity` is ever baked alongside u/v — four levels per facet. Not W1. | +| L6 | `3×(8:8:8:8)` SPOG quads | The Odoo factoring. Rejected outright. | +| L7 | `2×48` `hhtl ++ helix` absolute location | This IS location — and for a weather cell **location is the key, not the payload**. Storing it again in the value slab is the duplication §2's "never waste a slot" rule exists to prevent. (The `le-contract` §3 open item — whether the CANON key tail is literally an L7 facet — is untouched here and must not be unified silently.) | +| L8 | `helix ++ CAM_PQ` analog | Retired for this data by measurement, not taste: `weather-normalized-substrate.md` §12.12 **RETIRED the `Pair48` mint**, and §12.12's own regrade fixes the split — **rim = L1-metric carrier; azimuth = render carrier, deliberately not L1-metric**. Routing a weather scalar through the analog lane re-opens the exact "invented round-trip API" the helix crate names twice, unprompted. | +| G1/G2/G3 | wide carvings | **"New classes MUST NOT be born into G1–G3; the waiting room is not a destination."** The weather class is new. | + +**Chosen: L4 — `6 × (8:8)` palette256², one byte per ERA5 field, the pair being a physically +paired couple.** + +Justification in the catalogue's own terms: the classview is the **focus lens the data shape +wants**, and this data is *similarity data over commensurable scalars* — which is precisely +what L4 is for, and precisely what the arc has already measured. The pairing is not +decoration: `(u : v)` in one 8:8 pair means **wind-vector similarity is one 256×256 table +read**; `(msl : sp)`, `(T : q)` at a level, `(T_2m : T_sst)` are the natural couples. Slot +purity is preserved because *which* couple a pair holds is a ClassView fact (§3), never +inspectable from the bytes. + +### §2.3 ⚠ A stated deviation from L4's default reading — and the measurement that licenses it + +`le-contract.md` §3 pins L4's canonical distance/rank read to the **analytic Fisher-z codec** +(`bgz-tensor::fisher_z::{FamilyGamma, FisherZTable}`, certified ρ ≥ 0.999 on cosines). +**For weather scalars that is measured to be WRONG**, on this repo's own committed artifacts: + +- `weather-normalized-substrate.md` §12.1, real ERA5 `2m_temperature`, 1,038,240 gridpoints, + same 256 buckets, same percentile window: **linear MAE 0.0684 K / 0 empty buckets / 115.7 + effective buckets**, vs **Fisher-z-then-linear MAE 0.2168 K / 76 empty buckets / 28.1 + effective**. Fisher-z is **3.2× worse and burns 228 of 256 buckets**. Mechanism measured, not + assumed: `arctanh` moves resolution toward the bounds; ERA5 anomaly has **77 % of mass inside + |s| < 0.25** — mass in the middle. (§12.10 later narrowed this: scored against a noise floor + rather than round-trip, the two are indistinguishable; **what survives is bucket economy, not + a validity failure**. Bucket economy is exactly what a 1-byte-per-field scheme is spending.) +- `le-contract.md` §3 supplies the demarcation itself: *for a **monotone BOUNDED** continuous + field the exit is **analytic, not materialized** —* helix `RollingFloor` + (`quantize` → `bucket_center`), with elevation as the canonical case at **1 byte / RMSE ≈ + 0.11 % of range**. A weather scalar over a robust percentile window is exactly that shape. + +**Decision: the weather ClassView declares an L4 lane whose analytic codebook is the linear +`helix::quantize::RollingFloor`, not the Fisher-z gamma.** Distance is +`helix::distance::DistanceLut::from_floor` — 256 × 256 × u16 = **128 KB, L1/L2-resident**, with +the triangle-inequality regression already shipped (`distance.rs:87-105`, `:118-135`). The +deviation is recorded here in the plan, not smuggled into code, and the `le-contract` §3 note +is the authority for it — not an exception to it. + +**Two hard consequences, both falsifiable:** + +- **Circular variables must NEVER ride an L1 palette byte.** `end_idx` is monotone, so a + wrapped bearing puts 359° and 1° at *maximum* L1 distance (§12.2 point 5 — **stands**, and is + crate-documented as the "raw-azimuth 2π wrap"). Wind therefore rides as **(u, v) components, + two linear bytes**; a bearing, if ever wanted, is derived at read time from the pair. This is + also why `wind_speed` must be baked as its **own byte** and not reconstructed: §6.5 measured + stored speed ≥ `hypot(ū, v̄)` at **100 %** of samples, mean ratio **1.115**, max gap + **14.37 m/s** — the Jensen gap **is gustiness**, real signal. +- **The layout choice must not presuppose `D-WXA-5`'s outcome.** One byte per field is exactly + the quantisation whose fidelity the gate has never tested. Mitigation is structural, not + verbal: **W1 bakes a minimal justified field set, not all 122** (§2.5), the lane is + additive/reserve-don't-reclaim so it grows without a stride change, and `D-WXS-7`'s + graded-resolution ladder (16/64/256 levels) measures the marginal value of a byte *before* + 296 byte-positions are committed. + +### §2.4 The 14 statics — a separate class, a separate dataset, ONE version + +Time-invariant fields (orography family, `geopotential_at_surface`, `land_sea_mask`, +`lake_cover`/`lake_depth`, vegetation ×4, `soil_type`) baked into the versioned row would be +rewritten **92,044 times**: 1,038,240 × 14 B × 92,044 ≈ 1.3 PB of identical bytes. + +**Decision: statics get their own classid and their own dataset, baked once, ONE Lance +version, joined by the identical HEEL/HIP key.** The join is free precisely because the key is +the address. `D-WXS-5`'s bar is a version-count assertion — a bug that rewrites statics per +timestep is caught by `version_count == 1`, not by review. + +### §2.5 The W1 field set — minimal and justified, not all 122 + +Baking 122 fields in W1 would commit 122 byte-positions under an untested quantisation and +would make every downstream bar slower to run for no measured gain. W1 bakes exactly the +fields the open work needs: + +| facet | pair 0 | pair 1 | pair 2 | pair 3 | pair 4 | pair 5 | +|---|---|---|---|---|---|---| +| F0 (surface) | `msl` : `sp` | `u10` : `v10` | `wspd10` : `t2m` | `sst` : `tcwv` | `tcc` : `sic` | reserved-zero | +| F1 (850 hPa) | `u850` : `v850` | `t850` : `q850` | `z850` : `w850` | reserved | reserved | reserved | +| F2 (500 hPa) | `u500` : `v500` | `t500` : `q500` | `z500` : `w500` | reserved | reserved | reserved | + +`msl` + `u10/v10` are what `D-CZ-8` needs (ζ from the 10 m wind). `z500` is the classic +verification target and the field `weather-substrate-poc-v1`'s bake-off named. +Everything else is **reserved-zero**, dormant, expandable per class without a version bump +(`NodeRow` doctrine 1: *clean ⇒ expansion is classid-inherited*). + +### §2.6 Slot purity (§2) — how it is enforced, and where the temptation is + +**Nothing in the payload says what a byte means.** No variable names, no level numbers, no +column ordinals, no display labels. The mapping *"F1 pair 0 hi-byte = `v_component_of_wind` at +850 hPa, unit m/s, floor id 7"* lives in the **ClassView-side field manifest** selected by the +classview u16 — the register-file model: the SoA is dumb bytes, the class makes them +meaningful. + +Three places the temptation is real, and the ruling for each: + +1. **lat/lon.** It is a position, and §2 forbids positions in payload slots. It is **in the + key**, which is the key's job (`HEEL`/`HIP` are the cascade address; the OGAR pin is + explicitly *"the key prerenders nodes with zero value decode"*). Writing lat/lon into the + value slab as well would be both a §2 break and a duplication of the address. Forbidden. +2. **The codebook edges** `(lo, hi, floor_version)` per variable. Per-row they would cost + ~1 KB/row and would be a slot-purity break. They are **dataset (table) metadata**, aligned + to the Lance version boundary — which is exactly `weather-normalized-substrate.md` §4's + recommended operating rule: *calibrate once per epoch, freeze, align any re-roll with a + version boundary, record `(lo, hi, version)` in dataset metadata, so "same code ⇒ same + value" holds exactly within a Lance version and `QueryReference::at` rehydrates with the + floor that produced it.* The floor stamp and the Lance version become the same kind of + object: a calibration epoch. +3. **The timestamp.** It is the **version**, not a payload slot (§3). + +**Honest open item, carried not resolved:** `le-contract.md` §5 discrepancy 6 (via +`weather-normalized-substrate.md` §12.2 point 6, `[G-absence]`) — **there is no per-value-lane +*reading* selector in the contract at all.** `ReadMode` has three axes (tail / value_schema / +edge_codec); `ValueSchema` selects **presence**, not reading. So "the classview selects the L4 +reading" is doctrinally correct and **structurally unimplemented**. This plan does **not** +invent one. `D-WXS-1` ships the manifest as a **committed data artifact** the bake reads, with +its identity stamped into dataset metadata, and flags the missing selector as an +architecture-entropy item for the orchestrator. Anything else would be a contract-surface mint +smuggled in through a weather plan. + +--- + +## §3 The time axis — a timestep is a Lance version + +**Ruling in force:** `E-MARKOV-TEMPORAL-STREAM-1` — *episodic = Lance versions* (the OGAR +D-DELTA mapping, now primary); the ±5 window generalizes to a **version-range read** +(`QueryReference::at(v, rung)` + `deinterlace`) at any width, per-reader rung, replayable, and +**still a projection — zero copies**. + +| need | shipped surface (consumed, never re-implemented) | +|---|---| +| one timestep = one version (writer) | `graph::cycle_sink::LanceCycleWriter` (`open`/`bootstrap`/`head()`/`scan_image`/`reconcile_scans`), `VersionedGraph::commit_encounter_round` | +| time-travel read | `VersionedGraph::at_version(v)` (`versioned.rs:428`) · `current_version()` (`:435`) | +| version-range read | `planner::temporal::{QueryReference::at(v, rung), deinterlace, LanceVersion = u64, EpistemicMode, TemporalStatus, classify}` | +| per-key trajectory | `temporal::{local_trajectories, local_trajectory_of}` | +| diff two timesteps | `GraphDiff` (`versioned.rs:70`), `GraphSealStatus` (`:54`) | + +Two shipped tests already pin the invariant +(`a_whole_cycle_of_casts_is_one_wal_write_one_version`, +`p4a_drains_casts_and_seals_one_wal_write_one_version`). **Any weather deliverable that +re-implements a version writer is the defect, not the feature.** + +**What this replaces, concretely.** `substrate_comfort_d_cz_2_7.py:50` reads +`TIMESTEPS = [54358, 54358 + 4*182, 54358 + 4*365]` — three hand-typed integers with an +`EPOCH`-arithmetic anchor guard. On the substrate that is a **version range**, and the +19-storm set (each at its own `t0`) is a **version set**. The seasonal spacing that made those +three integers a real anti-cherry-pick control becomes a *stated version-selection policy* that +the harness can vary — which is what makes `C1`'s "≥3 independent timesteps" bar cheap to raise +to 30 instead of an argument about fetch budget. + +**Two honest risks, both pre-registered as bars (§4 W2):** + +- **92,044 versions on one dataset is unverified.** CONJECTURE. `D-WXS-6` measures manifest + growth and `at_version(1)` latency at N = 1 / 10 / 100 / 1000 versions and states whether the + growth is sub-linear. **KILL:** superlinear growth ⇒ "one version per timestep" does not + survive 65 years and the time axis needs an explicit chunking policy — a pre-registered + alternative, not a patch discovered mid-run. +- **The forecast/analysis two-lane model** (poc-v2 §1's *"one dataset, versions are cycles"*) + is untouched by this plan and stays as ruled. + +--- + +## §4 Waves, D-ids, and pre-registered bars + +**The rule for every bar below** (`CLAUDE.md` § The falsifiability rule): the bar is written +and **committed before the run**; it has a **control that can lose**; it has a **stay-silent +twin** using a **non-trivial** input (an empty-input silence case proves nothing); and it names +its **KILL** condition. An assertion implied by the code it tests is not a test. + +### W0 — decisions and pre-registration (no executable code) + +| D-id | deliverable | +|---|---| +| **D-WXS-0** | **classid mint** for the weather-cell class + the statics class. **BLOCKED** — OGAR-side, operator-gated. `0x0F = Geo` exists; the appid/classview half is open. **KILL:** if no domain:appid can be assigned, the bake must **refuse to write** — a dataset under `0x0000_0000` is unroutable and indistinguishable from a bootstrap row. Nothing downstream runs against a placeholder. | +| **D-WXS-1** | **the field manifest v1** — the ClassView-side table mapping (facet, pair, byte) → (ERA5 variable, level, unit, floor id). A committed data artifact, not literals in Rust. | +| **D-WXS-1a** | **the variable census, as a committed re-runnable probe** — the `.zmetadata` read behind §0.6. A census asserted in prose is the chat-only-figure defect, third instance in this arc. | + +**Bar B0 (manifest is load-bearing).** *Can-it-fire:* mutating one manifest entry must change +the bytes the bake writes for at least one cell. *Stay-silent twin (non-trivial):* editing a +**reserved-zero** slot's description, or reordering two entries that describe the same +(facet, pair, byte), must change **not one written byte**. **KILL:** if the bake's output is +insensitive to the manifest, the mapping is hard-coded somewhere and slot purity is theatre. + +### W1 — the bake (the missing Zarr → `NodeRow` path) + +| D-id | deliverable | file | +|---|---|---| +| **D-WXS-2** | key codec: `(lat_idx, lon_idx) ↔ NodeGuid` (HEEL/HIP, ragged tiles, lon-wrap range-**set**) | `crates/weather-poc/src/key.rs` | +| **D-WXS-3** | shared canonical floor calibration (per-variable `[lo,hi]` at the 0.4–99.6 percentile over a **global** sample, frozen per epoch, stamped into dataset metadata) | `.../src/floor.rs` | +| **D-WXS-4** | the bake: one timestep slab → 1,038,240 `NodeRow`s → **one Lance version** | `.../src/bake.rs` | +| **D-WXS-5** | statics bake — separate classid, separate dataset, one version | `.../src/statics.rs` | + +**Bar B1 (key).** *Primary:* all 1,038,240 `(lat,lon)` → key → `(lat,lon)` round-trip exactly, +**and no two distinct cells collide on one key**. *Control that can lose:* an off-by-one on +the **ragged last tile** (17 lat rows / 32 lon cols) must make the round-trip fail — if the +suite passes with that injected, it never tested the ragged case. *Stay-silent twin:* a box +entirely inside `[0°, 360°)` must return a **single** range and must **not** report a wrap; +a box crossing the seam must return **exactly two**. **KILL:** any collision ⇒ HEEL/HIP alone +does not address the grid and `TWIG` must be recruited — a design change, reported, not +patched. + +**Bar B2 (floor).** *Primary:* on the global calibration sample, **zero empty buckets** and +saturation below a pre-registered ε per variable (`RollingFloor::occupancy()` **is** the +histogram — no separate instrument). *Control that can lose:* a floor calibrated on **one 16° +box** applied globally must show high saturation — this is `GEO-DEGENERATE` re-homed, and +D-CZ-1 measured that construction saturating **72–97 %** per regime, so it is known-losable. +*Stay-silent twin:* the **global** floor must **not** show high saturation on any single box — +including the storm boxes, where the field is widest. **KILL:** if one global floor cannot +cover a variable within ε, that variable needs its own floor (§4 policy (c) hybrid) and the +"one comparable substrate" claim narrows *for that variable*, stated per-variable rather than +globally. + +**Bar B3 (bake round-trip).** *Primary:* read back version *v*, dequantize via +`bucket_center`, compare against the retained f32 truth — per-variable MAE within the +pre-registered ±½-bucket bound (≈ 0.195 % of span, `quantize.rs:11-18`). Plus the **512 B +verbatim** check (PR #907's compression-detection falsifier reused: the stride clears Lance's +256 B mini-block cutoff, takes the full-zip path, `1.00×`). *Control that can lose:* a bake +whose `floor_version` stamp is omitted must be **detected as unreadable**, never silently +mis-dequantized — this is the `quantize.rs:20-26` contract made into a test. *Stay-silent +twin:* a bake and re-read at the **same** floor version must produce **byte-identical** rows +across two runs (determinism). **KILL:** if the round-trip error exceeds the bucket bound, the +quantisation path has a bug and no fidelity number computed on it means anything. + +**Bar B4 (statics).** *Primary:* after committing N timestep versions to the dynamic dataset, +the statics dataset has **exactly one** version. *Control:* a deliberately re-writing statics +driver must trip it. *Stay-silent:* a legitimate one-off statics re-bake (a new epoch) must +produce exactly **two** and be labelled, not silently tolerated. + +### W2 — the time axis + +| D-id | deliverable | +|---|---| +| **D-WXS-6** | version-range read for a cell's time series: `QueryReference::at(v, rung)` + `deinterlace`; plus the version-scaling measurement (§3) | + +**Bar B5.** *Primary:* a 3-step series read by version range **byte-equals** the same 3 +versions read individually. *Control that can lose:* a `v_ref` outside the committed range must +return **empty**; a rung that excludes a row must **actually** exclude it (both halves — a +filter that never excludes and one that excludes everything carry identical information). +*Stay-silent twin:* a rung that should exclude **nothing** must return the full set — proving +the filter discriminates rather than always firing. **KILL:** as §3. + +### W3 — re-homing `D-WXA-5`, the real gate + +| D-id | deliverable | +|---|---| +| **D-WXS-7** | representation fidelity on the substrate: ρ(code-distance, field-distance) over pairs sampled from the **whole grid**, computed with `jc::reliability::spearman` | +| **D-WXS-8** | cross-variable comparability at grid scale — `p2_probe.py` re-run as a substrate read, ≥4 variables incl. ≥2 distinct units, ≥3 seasons | + +**Bar B6 — `D-WXA-5`, re-specified in three parts.** The poc-v2 bar (ρ ≥ 0.98 + shuffle +control fails) is **kept as the floor and demoted from the verdict**, per §0.3: + +- **(a) primary — ρ ≥ 0.9996**, the bar `p2_probe.py` actually pre-registers and where a real + pair **genuinely failed** (K×K = **0.999556**, below bar; the cross-unit pairs passed at + 0.999736 / 0.999722). A bar with a *demonstrated* failure on real data is a bar that can + fire. +- **(b) control that can lose — the shuffled decode table must score below 0.98.** Known + losable: D-CZ-1 measured `CAL-SHUFFLE` at ρ = **0.003 … 0.159** and RMSE **1498.6 Pa** + against real arms at 0.99999. This is the poc-v2 anti-vacuity control, retained verbatim. +- **(c) can-it-DIFFER — a graded-resolution ladder at 16 / 64 / 256 levels must produce a + strictly monotone ρ ordering, run BEFORE any verdict.** This is the half D-CZ-1 discovered + the hard way (§6.4: real-arm ρ spread 3×10⁻⁶…4.7×10⁻⁵ ⇒ *"C4 could not have fired"*). + **KILL for the metric:** if 16 levels is indistinguishable from 256, ρ is **decorative** on + this data and the gate moves to a **physical-unit** metric (per-variable MAE in K / Pa / + m·s⁻¹, range-normalised) *before* a single fidelity claim is made — exactly the amendment + D-CZ-1 made legitimately in preflight, and which would be illegitimate the moment a verdict + cell had been scored. +- **KILL for the substrate (the thesis-level one, unchanged from poc-v2):** every arm below + ρ ≈ 0.9 ⇒ the hierarchical-prefix-is-ancestry assumption does not survive contact with a + real physical field, and a large part of the substrate's `[H]`/`[S]` map downgrades from + "faithful code" to "useful router." That is a thesis result, not a weather result. + +**Bar B7 (`D-WXS-8`).** *Primary:* cross-unit ρ ≥ 0.9996 on the shared canonical floor. +*Control that can lose:* **per-variable floors must lose on cross-unit pairs** — measured +0.9997 shared vs **0.857–0.875** per-variable, a four-orders-of-magnitude discrimination that +appeared without being tuned for. *Stay-silent twin:* within-variable, shared must **not** +cost resolution — measured ≤ 0.0001 difference, zero empty buckets. **KILL:** if the shared +floor loses to per-variable floors on any cross-unit pair at grid scale, §4 policy (a) is +refuted and the substrate is a per-variable store — which would retract the arc's product +claim, and is the reason this runs early. + +### W4 — re-homing `D-CZ-8` (ζ + a range-normalised transfer metric) + +| D-id | deliverable | +|---|---| +| **D-WXS-9** | ζ = ∂v/∂x − ∂u/∂y as a **substrate read** — a 5-point stencil over neighbour keys; plus the quantisation falsifier below | +| **D-WXS-10** | the **vorticity regime ladder over the whole grid** (regimes = ζ-percentile bands, not four hand-picked boxes) + coverage-matched donor selection + a range-normalised `L` | + +**Why this is a substrate read.** A neighbour in HEEL/HIP address space is `lon_idx ± 1` / +`lat_idx ± 1` — **address arithmetic**, and within a HEEL tile the cells are contiguous. The +stencil that `grad_flat()` currently expresses as `np.gradient` over a Zarr chunk becomes a +key-range read. That is the whole point of §1. + +**Bar B8 — the falsifier this design *needs* and which does not exist anywhere in the arc: +differencing amplifies quantisation error.** ζ is a *difference of differences* of two +1-byte-quantised fields. A ±½-bucket error (≈0.195 % of span) is negligible on a value and can +be **O(1) relative** on a small gradient. *Primary:* ρ(ζ from quantised u,v ; ζ from retained +f32 truth) ≥ a pre-registered bar, reported **per ζ-magnitude decile** (the error will be +worst where ζ is smallest, and a global ρ would hide exactly that). *Control that can lose:* +ζ derived from a **16-level** palette must be visibly worse. *Stay-silent twin:* on a laminar, +near-zero-ζ region the discriminator must **not** flag turbulence — the `closed_class_guess` +150/150 defect in its weather clothing. **KILL:** if quantised ζ does not track true ζ, +**ζ must be baked as its own derived lane** rather than derived at read time — a real design +fork, pre-registered here so it is a decision and not a mid-run rescue. + +**Bar B9 — the range confound, and exactly what the full grid does and does not buy.** + +*What the full grid does NOT buy:* it **does not dissolve §7.9's confound.** `L` was measured +to be essentially a function of **coverage** — `L` vs cell `saturation` at Pearson **+0.917** / +Spearman **+0.833**, and `L̄[T]` vs the regime's own value range at **Spearman +1.000, +perfectly monotone** (R4's implied range ≈ 18× R1's: ~7075 Pa vs ~386 Pa). A foreign fixed +codebook applied to a wider target will saturate more, on 1.04 M cells exactly as on 4225. +Moving to the full grid changes the sample, not the arithmetic. **This plan asserts no +dissolution.** + +*What the full grid DOES buy — three specific, non-rhetorical things:* + +1. **Coverage becomes controllable.** With 1,038,240 cells you can **select** donor and target + populations **matched on value range and offset** by construction. With four hand-picked + boxes you could only observe the confound; with the grid you can hold it constant — which + is the methodological frame (hold variables constant to test the others) applied to the + confound instead of to the regime. +2. **"Donor" can finally mean a *regime* rather than a *range*.** The shared canonical z-floor + (`D-WXS-8`, measured cross-unit ρ 0.9997 vs 0.857–0.875 per-variable) puts every variable on + one commensurable scale, so a donor codebook can be defined by a **ζ-band** rather than by + whatever range four boxes happened to have. +3. **The regime axis stops being n = 4.** ζ-percentile bands over the globe give a regime + sample of millions of cells per band and let `C1`'s "≥3 timesteps" bar become ≥30 without a + fetch-budget argument. It also removes `C5`'s structural blocker's *cause* — though **not** + `C5` itself: the golden index floor needs N ≥ F(17)² = **2,550,409** and the global grid has + **1,038,240** cells, so `GEO-GOLDEN-HI` is **still not constructible**, now short by ~2.5× + rather than by three orders of magnitude. Stated so nobody re-discovers it as a surprise. + +*The bar itself.* **Primary:** the ζ-regime effect on a **range-normalised** `L` must survive +**coverage matching** — report `L` both raw and residualised on `saturation`, and report the +matched-population version alongside. **Control that can lose:** a regime ladder built on a +**shuffled ζ field** (same marginal distribution, destroyed spatial structure) must show **no** +monotone `L̄` trend. **Stay-silent twin:** two donor/target sets drawn from the **same** ζ-band +must show `L̄ ≈ 0` — if band-internal transfer is as costly as cross-band transfer, the bands +are not a regime axis at all. **KILL:** if `L̄` still rank-correlates ρ ≈ 1.0 with range *after* +coverage matching, then the transfer metric **is** a coverage metric, `D-CZ-8`'s premise fails, +and the result is reported as such — **no further normalisation is attempted until it agrees.** +Re-normalising a metric until it stops showing the confound is how a confound becomes a finding. + +### W5 — gated on W3 green: the honest tail + +| D-id | deliverable | +|---|---| +| **D-WXS-11** | **measure** a full-grid bake wall time. The artifact must state the ~8.1 s serial extrapolation as the **prior** and either confirm or correct it in the same document (§0.5). No claim about parallelism — `D-KIA-A2` owns that. | +| **D-WXS-12** | `jc` ↔ `ndarray` reliability agreement — poc-v2 `D-WXB-4` carried over unchanged: identical non-degenerate inputs must agree; the degenerate case must be **reported** (`jc` → `None`), never folded as `0.0`. **`jc` is the authority; `ndarray::hpc` is the SIMD-side mirror.** | + +**Sequencing.** `W0 → W1 → W2 → W3 → (W4, W5)`. **Nothing in W4 or W5 runs before W3's gate +reports.** A negative W3 is the most informative result available and is thesis-relevant rather +than weather-relevant. + +--- + +## §5 The Python probes, classified: substrate read vs numpy function + +Per the brief, every computation in `substrate_comfort_d_cz_0_1.py` and +`substrate_comfort_d_cz_2_7.py` is classified. **"Substrate"** = a shipped primitive exists and +the Python is re-deriving it; **"numpy"** = no substrate equivalent exists and the Python is +correct to be Python; **"probe-side over a substrate read"** = the statistic stays a statistic +but its *input* becomes a key-range scan instead of a Zarr chunk. + +| Python | classification | substrate surface / note | +|---|---|---| +| `encode_decode(values, lo, hi)` — 256 uniform levels, clip-saturate, decode to level centre | **SUBSTRATE** | `helix::quantize::RollingFloor::{uniform, quantize, bucket_center}` — the *identical* linear formula (`quantize.rs:99-108`, `:248-250`). `CAL-ABS` **is** a `RollingFloor` with a donor's bounds. This is the single largest re-derivation in the arc. | +| `cell_metrics` → `occupancy`, `saturation` | **SUBSTRATE** | `RollingFloor::occupancy()` **is** the histogram — *"no separate histogram"* (`quantize.rs:5-9`); saturation = the two rim buckets' share; `drift_score()` is the regime-change telltale, free. | +| `spearman(a,b)` (tie-averaged, `nan` on constant input) | **SUBSTRATE** | `jc::reliability::spearman(&[f64], &[f64]) -> Option` — same tie semantics, and `None` where Python returns `nan`. `jc` is the operator-named authority. | +| `grad_flat` (`np.gradient`, Pa/cell, **no cos(lat)**) | **SUBSTRATE-ADJACENT** — becomes a neighbour-key stencil | ⚠ Carry **both** definitions, labelled with units: the committed ladder used the **flat, no-cos** definition (D-CZ-0 §6.2 identified it from the data at max\|ratio−1\| = 0.069), and it **understates the zonal gradient by 1/cos(lat)** — R3 at 60 N is ~40 % low. Silently switching to the metric-correct definition would break comparability with every committed figure. | +| `box_index(clat, clon)` + `% len(LONS)` | **SUBSTRATE** | HEEL-prefix **range-set** (§1.2/§1.3). The `% 1440` is the seam split. | +| `rank_codec` (256 **quantile** levels in-window) | **NUMPY — no substrate carrier exists** | `RollingFloor` is *linear over a window*, not quantile-spaced. `CAL-RANK` has no shipped Rust equivalent. Either it stays the Python reference arm, or it is a **new codec** owing its own probe. Do not paper over this by calling `RollingFloor` "the rank codec". | +| `fisherz_codec` | **SUBSTRATE** (`helix::fisher_z` / `bgz-tensor::fisher_z`) — but **control-arm only** | §12.1 measured it 3.2× worse and 228/256 buckets burnt on these shapes; §12.10 narrowed that to *bucket economy, not validity*. Keep it as the shape-mismatch control, never as the incumbent. | +| `decay_length_cells`, `gini`, `tail_ratio` (C1c instruments) | **NUMPY** — no substrate equivalent (`jc::stats` has none) | Stay probe-side statistics, but computed **over a substrate read** (a key-range scan), not over a Zarr chunk. | +| `fetch(var, key)` + numcodecs blosc decode | **PYTHON, correctly** | poc-v2 §1's Stage-A **disposable** ingest. Do not gold-plate the throwaway; do not put GRIB2 or a Rust Zarr reader on the critical path. | +| `t_index()` + the `EPOCH` anchor assert | **SUPERSEDED** | The time axis is Lance versioning (§3). The anchor guard's *spirit* survives as a manifest/metadata check that fails loudly if the store is re-chunked. | +| `TIMESTEPS = [...]`, per-storm `t0` | **SUPERSEDED** | A version range and a version set (§3). | + +--- + +## §6 Work split — design/judgment vs mechanical grindwork + +### §6.1 Design / judgment (Opus or the main thread — accumulation, never delegated) + +- The **facet-layout choice** and its stated deviation from L4's default Fisher-z reading + (§2.2/§2.3) — it holds two measured findings and a doctrine note in mind at once. +- The **key assignment** and the three wrinkles (§1) — a wrong tiling is a silent + data-corruption class, not a bug. +- **Every bar in §4** — a bar written by the code's author is the + `E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` rule waiting to fire. Bars + are pre-registered by design, committed before the run, and (for any verdict-tier claim) + independently spec-audited. +- The **`D-WXA-5` re-specification** (§0.3 / bar B6) — a judgment call about an instrument's + dynamic range, grounded in a committed measurement. +- The **§7.9 confound treatment** (bar B9) — deciding what the grid buys and what it does not. +- **`D-WXS-0`** (the classid mint) — operator/OGAR-gated, not a worker task. + +### §6.2 Mechanical grindwork (Sonnet workers, one file each, disjoint) + +Every worker brief pastes `.claude/v3/knowledge/sonnet-worker-guardrails.md` §1 **verbatim**, +plus §2 (vocabulary disambiguation) and §5 (STOP+report triggers). No grindwork spawn without +it. Every brief also carries: *read `.claude/board/AGENT_LOG.md` before starting; do NOT write +it — leave your record in your own tag-file under `probes/weather-p1/exec-runs/`; the +orchestrator consolidates.* + +| worker | file (sole owner) | task shape | +|---|---|---| +| W-key | `crates/weather-poc/src/key.rs` | encode/decode + range-set; bar B1's tests | +| W-floor | `crates/weather-poc/src/floor.rs` | drive `RollingFloor` calibration from a sample; emit `(lo,hi,version)`; bar B2 | +| W-lane | `crates/weather-poc/src/lane.rs` | L4 pack/unpack **given** the manifest; zero domain knowledge | +| W-bake | `crates/weather-poc/src/bake.rs` | slab → `NodeRow`s → `LanceCycleWriter`; bar B3 | +| W-statics | `crates/weather-poc/src/statics.rs` | the one-version statics path; bar B4 | +| W-stencil | `crates/weather-poc/src/stencil.rs` | neighbour-key arithmetic + ζ; bar B8 | +| W-metrics | `crates/weather-poc/src/metrics.rs` | `jc` battery wiring; bars B6/B7 | +| W-tests | `crates/weather-poc/tests/.rs` | one file per bar, never shared | + +**Orchestrator-only files (shared — a worker touching one is the defect):** +`crates/weather-poc/src/lib.rs` (the `mod` lines), `crates/weather-poc/Cargo.toml`, the root +`Cargo.toml` (**not touched at all** — `weather-poc` is workspace-**EXCLUDED**, on the same +pattern as `jc`, `helix`, `sigker`, `perturbation-sim`), +`crates/lance-graph-contract/src/canonical_node.rs` (the `ValueTenant` enum + `VALUE_TENANTS` — +a canon act, Opus/orchestrator only, and note discriminant **15 is reserved for +`BoardAggregates`**; a weather tenant takes **16**, with its offset **derived** via +`value_offset()` and never written as a literal), and every `.claude/board/*` file. + +### §6.3 Crate scaffold constraints (non-negotiable) + +- **No new repositories.** `crates/weather-poc`, workspace-**EXCLUDED**, on the + `perturbation-sim` template. Verified via + `cargo test --manifest-path crates/weather-poc/Cargo.toml`. +- **Zero-dep default build.** Any `ndarray` dependency is **optional, behind an off-by-default + feature, and sourced by GIT URL — never `path`** (an optional *path* dep is read at manifest + resolution and breaks a clean checkout even with the feature off: codex P2 on #504, the + helix-#460 lesson, recorded verbatim in `perturbation-sim/Cargo.toml`). +- **SIMD only via `ndarray::simd::*`** (`ndarray/src/lib.rs:241`, `src/simd.rs`). **Never + `ndarray::hpc::*` from a consumer**, and never raw intrinsics. +- **The lance family is upstream-authoritative and moves in exact lockstep.** No version bump + is proposed by this plan: `lance = "=9.0.0"` family, `lancedb = "=0.33.0"`, `arrow 58`, + `datafusion 54` **and** `53` (both required — `deltalake-core 0.32.4` pins 53 upstream; + collapsing the lock to one breaks the `delta` feature), rust `1.97.1`. +- **Credentials by name only** — `dev_s3_env::s3_options()` reads the set and returns `None` + when any is missing; on a remote path that must be a **hard error**, never a silent local + fallback. **S3 hydrates; a local mmap-capable directory stores** (#901); a network mount that + looks local is the named trap. + +--- + +## §7 Honest grading + +| claim | grade | provenance | +|---|---|---| +| 1,038,240 cells × 512 B = 0.495 GiB/timestep | `[G]` | arithmetic on the const-asserted stride | +| 122 fields fit one 512 B row at 1 B/field | `[G]` | arithmetic against the **292 B** free budget (§0.4), not 384 | +| 122 fields do **not** fit as 4+12 facets at 2 B/field | `[G]` | 21 × 16 = 336 > 292 | +| HEEL/HIP addresses 721×1440 with 5–6 of 12 quaternary levels/axis | `[G]` | 4⁵ ≥ 721, 4⁶ ≥ 1440 | +| the axes tile raggedly (721 = 11×64+17; 1440/64 = 22.5) | `[G]` | arithmetic | +| shared canonical z-floor beats per-variable floors cross-unit (0.9997 vs 0.857–0.875) | `[H]` | `p2_probe.py`, one timestep, 3 variables, §12.8 — graded `[H]` there pending seasons + a 4th unit | +| the K×K pair (0.999556) is **below** the 0.9996 bar | `[G]` | measured, corrected under review on #920 | +| Fisher-z burns 228/256 buckets on ERA5 temperature | `[G]` on the buckets; `[H]` on the verdict | §12.1 measured; §12.10 narrowed to bucket economy, not validity | +| ρ saturates on the diagonal (real-arm spread 3e-6…4.7e-5) | `[G]` | D-CZ-1, `substrate_comfort_d_cz_0_1.json` | +| `L` vs `saturation` Pearson +0.917; `L̄` vs range Spearman +1.000 | `[G]` | §7.9, committed JSON | +| **`D-WXA-5`'s ρ ≥ 0.98 bar is at risk of being vacuous** | **`[H]` — inference from the two rows above; the reason W3 part (c) exists** | §0.3 | +| ~8.1 s per full-grid timestep | **CONJECTURE** — serial extrapolation from ONE measured cycle (514 ms, 65,536 owners); `D-KIA-A2` Queued/unbuilt; and a bake is not a mailbox cycle | §0.5 | +| 92,044 Lance versions on one dataset is viable | **CONJECTURE** — unmeasured; `D-WXS-6` is the falsifier | §3 | +| L4 + linear `RollingFloor` is the right lane for weather scalars | `[H]` — composed from measured pieces, unrun **as a lane** | §2.2/§2.3 | +| ζ derived from 1-byte-quantised u,v is usable | **CONJECTURE — and the plan's most likely failure point** | bar B8 | +| the grid dissolves the §7.9 confound | **FALSE, asserted nowhere** — it buys *control*, not dissolution | bar B9 | +| `GEO-GOLDEN-HI` becomes constructible at grid scale | **FALSE** — needs 2,550,409; grid has 1,038,240 | §4 W4 | + +--- + +## §8 What contradicts the commissioning framing (consolidated) + +1. **`D-WXA-5` is not on `STATUS_BOARD.md`; no `D-WXA-*` row is** (§0.1). It lives only in the + plan file and the `INTEGRATION_PLANS` prose. +2. **The "C2 gate" label belongs to a different plan** (§0.2). +3. **`D-WXA-5` as written is probably unable to fail** on this data (§0.3) — the arc's own + committed measurement says so. +4. **"32 facets / 33.2 M facets per timestep" overstates the usable budget by ~78 %** — the + real free budget is 292 bytes / 18 facets (§0.4). +5. **The ~8.1 s figure is both an unverified extrapolation and a category error** — a bake is a + columnar write, not 16 mailbox seal cycles (§0.5). +6. **`crates/weather-poc` does not exist** — confirmed against the crate listing; and + `weather-substrate-poc-v2.md` indeed specifies **no bake step**, which is the direct cause + of the arc staying in Python. +7. **`helix`/`Signed360` is not a wind carrier and the `Pair48` mint is RETIRED** (§12.12) — + any design that reaches for `HelixResidue` for weather is re-opening a four-corrections-deep + dead end. This plan does not. +8. **There is no per-value-lane *reading* selector in the contract** (`[G-absence]`, §2.6) — so + "the classview selects the L4 reading" is doctrine without a mechanism. Flagged, not + invented around. + +--- + +## §9 `weather-substrate-poc-v2.md`, part by part + +| part | verdict | +|---|---| +| §1 disposable Stage-A vs permanent Stage-C ingest split | **SURVIVES** — adopted verbatim | +| §1 "versioning is CONSUMED, not built" | **SURVIVES** — §3 consumes exactly that surface | +| §1 one dataset, versions are cycles | **SURVIVES** | +| §1 S3 hydrates / local mmap stores (#901) | **SURVIVES** | +| §2 `D-WXA-1` Stage-A ingest | **SURVIVES**, re-scoped: emit the **shared `soa:*` metadata block** and the field manifest, not just a flat slab | +| §2 `D-WXA-2` `crates/weather-poc` scaffold, workspace-EXCLUDED | **SURVIVES — and is `D-WXS-4`'s prerequisite. NEVER BUILT.** | +| §2 `D-WXA-3` encoder arms A–E | **SUPERSEDED in part.** Arms A/B (helix48, helix+residue) are retired for scalars by §12.1/§12.12. Arm E (`cascade_key`) is **not an encoder arm at all** — it is the **key design** (§1). What remains a live bake-off is **arm D (bgz17 hierarchical palette) vs the measured incumbent (linear `RollingFloor` + `DistanceLut`)**, which is also `F-1`. | +| §2 `D-WXA-4` the `jc` battery | **SURVIVES** — `D-WXS-7`/`D-WXS-8` | +| §2 `D-WXA-5` the gate | **SURVIVES as the floor, re-specified as the verdict** (§0.3, bar B6). **NEVER RUN.** | +| §3 `D-WXB-1..4` ndarray parity / no-silent-scalar / throughput / jc↔ndarray divergence | **SURVIVES**, deferred to W5 (`D-WXS-11`, `D-WXS-12`). **NEVER BUILT.** | +| §4 `D-WXC-1..5` permanent ingest, two-stage retrieval, external score, three lanes | **SURVIVES as future work**, unchanged and still gated behind the representation gate. **NEVER BUILT.** | +| §5 repos / zero new repositories | **SURVIVES** | +| §6 pins (incl. the ⊘ double correction on datafusion 54 **and** 53) | **SURVIVES** | +| §7 credentials | **SURVIVES** | +| §0 C1/C2/⊘C3 (GRIB2 gone; `ecmwf-opendata` is Phase C; the invented Zarr object name) | **SURVIVES**, and ⊘C3's apparatus lesson — *a title match is not an existence check; list the bucket* — is the reason `D-WXS-1a` commits the census as a probe | + +**Net:** poc-v2's *structure* survives almost entirely. What it was missing is the thing this +plan adds — **the bake** — and what it got wrong is one bar's threshold, which the arc's own +later measurement exposed. + +--- + +## §10 Cross-references + +`.claude/v3/soa_layout/le-contract.md` §2/§3/§3a/§5 · `CLAUDE.md` § CANON — Minimal SoA node, +§ The falsifiability rule, § Mandatory Board-Hygiene Rule, § Key Dependencies · +`/home/user/OGAR/CLAUDE.md` § Tier interpretation — 256×256 CENTROID TILE, § the 3×4 path · +`.claude/plans/weather-substrate-poc-v2.md` · `.claude/plans/substrate-comfort-zones-v1.md` +§7.8/§7.9 · `.claude/knowledge/weather-normalized-substrate.md` §3/§4/§5/§8/§9/§12.1/§12.2/ +§12.6/§12.8/§12.10/§12.12 · `.claude/knowledge/encoding-ecosystem.md` (mandatory, read) · +`.claude/knowledge/helix-cartesian-vs-fisher2z.md` (the doctrine §12.12 was retired for not +reading) · `.claude/knowledge/s3-hydration-lifecycle.md` · +`.claude/v3/knowledge/sonnet-worker-guardrails.md` · `.claude/board/EPIPHANIES.md` +`E-MARKOV-TEMPORAL-STREAM-1`, `E-V3-FACET-4-PLUS-12`, `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`, +`E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1`, +`E-ZERO-FOR-ELEVEN-THE-AUTHOR-CANNOT-AUDIT-HIS-OWN-FALSIFIERS-1` · +`.claude/board/AGENT_LOG.md` 2026-08-05 (PROBE-IGNITION-64K) · +iron rules `I-VSA-IDENTITIES`, `I-NOISE-FLOOR-JIRAK`, `I-LEGACY-API-FEATURE-GATED`. + +--- + +## §11 Board rows landed with this plan + +> Written by the orchestrator (sole writer of `.claude/board/*`) in the same commit as this +> plan, per the Mandatory Board-Hygiene Rule. §11.1 is the `STATUS_BOARD.md` block; §11.2 is +> the `INTEGRATION_PLANS.md` entry. + +### §11.1 `STATUS_BOARD.md` — the D-WXS block + +Landed as its own section. Every row `Queued` except `D-WXS-0` (**Blocked**, operator/OGAR). +Carries two standing notes: the poc-v2 ladder's board-hygiene gap (`grep -c WXA` = 0, nothing +ever built), and that the global grid does **not** unblock `GEO-GOLDEN-HI` (needs 2,550,409; +grid has 1,038,240). + +### §11.2 `INTEGRATION_PLANS.md` — PREPEND entry + +Landed with the plan, recording: the bake as the closed gap, the HEEL/HIP key design and its +three wrinkles, the L4 + linear-`RollingFloor` lane with its stated deviation, the corrected +292-byte capacity budget, statics-as-own-class, timestep-as-version, the `D-WXA-5` +re-specification, and the §7.9 position (the grid buys control, not dissolution). diff --git a/Cargo.toml b/Cargo.toml index 0c3846a9b..2ea11c1c5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -109,6 +109,13 @@ exclude = [ # crates/onebrc-probe/README.md. Verify via # `cargo test --manifest-path crates/onebrc-probe/Cargo.toml`. "crates/onebrc-probe", + # ERA5 0.25-degree global grid baked onto the canonical 512-byte NodeRow -- + # one cell = one node, one timestep = one Lance version. Standalone and + # genuinely zero-dep (see the crate manifest for why helix is NOT an + # optional path dep). Verify via + # `cargo test --manifest-path crates/weather-poc/Cargo.toml`. + # Plan: .claude/plans/weather-soa-bake-v1.md. + "crates/weather-poc", ] resolver = "2" diff --git a/crates/weather-poc/Cargo.lock b/crates/weather-poc/Cargo.lock new file mode 100644 index 000000000..fdaedaca0 --- /dev/null +++ b/crates/weather-poc/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "weather-poc" +version = "0.1.0" diff --git a/crates/weather-poc/Cargo.toml b/crates/weather-poc/Cargo.toml new file mode 100644 index 000000000..681d9acf2 --- /dev/null +++ b/crates/weather-poc/Cargo.toml @@ -0,0 +1,40 @@ +[package] +name = "weather-poc" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" +publish = false +description = "Zarr -> NodeRow: the ERA5 0.25-degree global grid baked onto the canonical 512-byte SoA row. One cell = one node (HEEL = 16-degree tile, HIP = within-tile), one timestep = one Lance version. Plan: .claude/plans/weather-soa-bake-v1.md" + +# Standalone research crate, EXCLUDED from the lance-graph workspace (same +# pattern as jc / sigker / helix / perturbation-sim). Verify with: +# cargo test --manifest-path crates/weather-poc/Cargo.toml +# +# ── ZERO-DEP BY CONSTRUCTION (an orchestrator decision, plan §6.1) ─────────── +# This crate has NO dependencies, not even optional ones, and that is +# deliberate rather than incidental: +# +# * `helix::quantize::RollingFloor` is the shipped carrier for the linear +# 256-level floor this crate needs (plan §5 classifies the arc's Python +# `encode_decode` as a re-derivation of exactly it). But `helix` carries a +# MANDATORY ndarray git dep, and an *optional path* dep is read at manifest +# resolution -- so merely offering `helix = { path = "../helix", optional }` +# would break a clean checkout even with the feature off. That is the +# codex-P2 trap on #504 / the helix-#460 lesson, named in plan §6.3. +# * So `floor.rs` carries the linear formula itself, documented as identical +# to helix's, and the AGREEMENT between the two is a separate deliverable +# run across manifests -- the same shape as D-WXS-12 (jc <-> ndarray), where +# parity is a measured comparison rather than a dependency edge. +# +# If a future wave needs `ndarray::simd`, add it the way perturbation-sim does: +# optional, behind an off-by-default feature, sourced by GIT URL -- never a path. +# All SIMD comes from `ndarray::simd` per the workspace rule, never raw +# intrinsics, and never `ndarray::hpc::*` from a consumer. + +[dependencies] + +[dev-dependencies] + +[lib] +name = "weather_poc" +path = "src/lib.rs" diff --git a/crates/weather-poc/data/field_manifest_v1.tsv b/crates/weather-poc/data/field_manifest_v1.tsv new file mode 100644 index 000000000..cc1dac4c3 --- /dev/null +++ b/crates/weather-poc/data/field_manifest_v1.tsv @@ -0,0 +1,43 @@ +# field_manifest_v1.tsv -- the ClassView-side field manifest for weather-poc +# W1 (plan .claude/plans/weather-soa-bake-v1.md §2.5/§2.6/D-WXS-1). +# +# One row per OCCUPIED byte of the L4 `6x(8:8)` facet layout +# (.claude/v3/soa_layout/le-contract.md §2/§3). Reserved-zero slots are NOT +# rows (plan §2.5) -- their absence IS their meaning: dormant, expandable +# later without a layout change (RESERVE, DON'T RECLAIM). +# +# lo/hi convention: within a pair written "A : B" in the plan §2.5 table, +# A is the LOW byte and B is the HIGH byte -- the pair's own left-to-right +# order. Kept consistent across every row below. +# +# facet: 0 = F0 (surface), 1 = F1 (850 hPa), 2 = F2 (500 hPa) -- plan §2.5. +# variable: the ERA5 variable name exactly as it appears in the Zarr store +# (verified against a live .zmetadata read, plan §0.6). +# level_hpa: the pressure level in hPa, or the literal token `surface`. +# floor_id: a stable identifier for the calibration floor this field uses; +# two rows sharing a floor_id share the same [lo, hi] calibration (none do +# in this committed manifest -- each field here is calibrated on its own +# physical range, plan §2.3/D-WXS-3). +facet pair byte variable level_hpa unit floor_id +0 0 lo mean_sea_level_pressure surface Pa msl +0 0 hi surface_pressure surface Pa sp +0 1 lo 10m_u_component_of_wind surface m s-1 u10 +0 1 hi 10m_v_component_of_wind surface m s-1 v10 +0 2 lo 10m_wind_speed surface m s-1 wspd10 +0 2 hi 2m_temperature surface K t2m +0 3 lo sea_surface_temperature surface K sst +0 3 hi total_column_water_vapour surface kg m-2 tcwv +0 4 lo total_cloud_cover surface 1 tcc +0 4 hi sea_ice_cover surface 1 sic +1 0 lo u_component_of_wind 850 m s-1 u850 +1 0 hi v_component_of_wind 850 m s-1 v850 +1 1 lo temperature 850 K t850 +1 1 hi specific_humidity 850 kg kg-1 q850 +1 2 lo geopotential 850 m2 s-2 z850 +1 2 hi vertical_velocity 850 Pa s-1 w850 +2 0 lo u_component_of_wind 500 m s-1 u500 +2 0 hi v_component_of_wind 500 m s-1 v500 +2 1 lo temperature 500 K t500 +2 1 hi specific_humidity 500 kg kg-1 q500 +2 2 lo geopotential 500 m2 s-2 z500 +2 2 hi vertical_velocity 500 Pa s-1 w500 diff --git a/crates/weather-poc/src/floor.rs b/crates/weather-poc/src/floor.rs new file mode 100644 index 000000000..03347f72c --- /dev/null +++ b/crates/weather-poc/src/floor.rs @@ -0,0 +1,468 @@ +//! `D-WXS-3` — the shared canonical floor calibration + the linear 256-level +//! quantiser (`.claude/plans/weather-soa-bake-v1.md` §4 W1, bar B2). +//! +//! Two halves, one genuinely new, one a documented re-expression: +//! +//! 1. **Calibration (new work).** [`calibrate`] derives a floor window +//! `[lo, hi]` from a sample of `f64` values at the robust +//! **0.4th – 99.6th percentile** ([`LO_PERCENTILE`] / [`HI_PERCENTILE`]) +//! — a window that deliberately excludes the extreme tails so a single +//! outlier sample cannot collapse `[lo, hi]` toward a degenerate span — +//! and stamps the result with a [`CalibratedFloor::floor_version`] +//! identifying the calibration epoch that produced it. +//! 2. **The linear floor (a re-expression, not a re-derivation).** +//! [`CalibratedFloor::quantize`] and [`CalibratedFloor::bucket_center`] +//! are byte-for-byte the same arithmetic as +//! `helix::quantize::RollingFloor::{quantize, bucket_center}` +//! (`crates/helix/src/quantize.rs:99-108`, `:248-250`): 256 uniform +//! levels over `[lo, hi]`, clip-saturating outside the window, decoding +//! to the bucket centre. This crate is zero-dep BY CONSTRUCTION (see +//! `weather-poc/Cargo.toml`'s ZERO-DEP-BY-CONSTRUCTION note — an +//! *optional path* dep on `helix` would still be read at manifest +//! resolution and break a clean checkout), so the formula is +//! re-expressed here rather than imported, and the two implementations +//! must be kept in lockstep by inspection, not by a shared dependency +//! edge — the same shape as `D-WXS-12` (`jc` ↔ `ndarray::hpc` +//! agreement): parity is a measured comparison, not a dependency. +//! +//! [`CalibratedFloor::occupancy`] **is** the histogram — no separate +//! instrument is kept, mirroring the property +//! `helix::quantize::RollingFloor`'s own module docs state +//! (`crates/helix/src/quantize.rs:5-9`). + +/// Number of quantisation buckets. Mirrors `helix::constants::PALETTE_SIZE` +/// (`= 256`), duplicated here rather than imported — see the module docs. +const BUCKETS: usize = 256; + +/// Lower percentile of the robust calibration window (0.4th percentile). +/// +/// Chosen — per the deliverable spec — to exclude the extreme tail so a +/// single outlier sample cannot collapse `[lo, hi]` to a degenerate or +/// near-degenerate span. See [`HI_PERCENTILE`] for the upper bound and the +/// module docs for the full window. +pub const LO_PERCENTILE: f64 = 0.4; + +/// Upper percentile of the robust calibration window (99.6th percentile). +/// See [`LO_PERCENTILE`]. +pub const HI_PERCENTILE: f64 = 99.6; + +/// A calibrated, versioned linear 256-bucket floor over `[lo, hi]`. +/// +/// Produced only by [`calibrate`] — there is no public constructor that +/// bypasses calibration, because [`Self::floor_version`] is a promise about +/// *how* `(lo, hi)` was derived, and a hand-built floor would break that +/// promise silently. +/// +/// The quantise/dequantise arithmetic ([`Self::quantize`], +/// [`Self::bucket_center`]) is the re-expressed half of `D-WXS-3` — see the +/// module docs for the exact `helix::quantize::RollingFloor` line +/// references it mirrors. +#[derive(Debug, Clone)] +pub struct CalibratedFloor { + lo: f64, + hi: f64, + floor_version: u64, + occupancy: [u32; BUCKETS], + samples: u64, +} + +impl CalibratedFloor { + /// **COMPUTE — `&self`.** Map `value` to a bucket in `0..=255`. + /// + /// Identical formula to `helix::quantize::RollingFloor::quantize` + /// (`crates/helix/src/quantize.rs:99-108`): + /// + /// - Returns `0` for degenerate bounds (`hi <= lo`). + /// - Saturates to bucket `0` for `value <= lo` and to bucket `255` for + /// `value >= hi`. + /// - Otherwise `idx = floor(((value - lo) / (hi - lo)) * 256)`, clamped + /// to `[0, 255]`. + pub fn quantize(&self, value: f64) -> u8 { + if self.hi <= self.lo { + return 0; + } + let t = (value - self.lo) / (self.hi - self.lo); + let idx = (t * 256.0).floor(); + let idx = idx.clamp(0.0, 255.0); + idx as u8 + } + + /// **COMPUTE — `&self`.** The representative centre value of bucket `b`. + /// + /// Identical formula to `helix::quantize::RollingFloor::bucket_center` + /// (`crates/helix/src/quantize.rs:248-250`): + /// `lo + ((b + 0.5) / 256) * (hi - lo)`. + pub fn bucket_center(&self, b: u8) -> f64 { + self.lo + ((b as f64 + 0.5) / BUCKETS as f64) * (self.hi - self.lo) + } + + /// Decode bucket `b`, but only when `floor_version` matches this + /// floor's own [`Self::floor_version`]. + /// + /// Returns `None` on a version mismatch instead of silently returning a + /// bucket centre computed from the WRONG `[lo, hi]` window. + /// `helix::quantize::RollingFloor`'s module docs state the same + /// contract for its own version stamp (`crates/helix/src/quantize.rs: + /// 20-26`): "same value → same `u8`" holds only within a stable floor + /// version, so decoding across a version boundary without checking is + /// exactly the silent-mis-dequantisation failure this method closes. + pub fn decode(&self, b: u8, floor_version: u64) -> Option { + if floor_version != self.floor_version { + return None; + } + Some(self.bucket_center(b)) + } + + /// The calibration-epoch stamp. + /// + /// A deterministic FNV-1a-64 hash (see the private `fnv1a_64` helper in + /// this module) of: `lo`, `hi` (the calibration OUTPUT), the + /// calibration sample's finite-value count, and [`LO_PERCENTILE`] / + /// [`HI_PERCENTILE`] (the calibration PARAMETERS). Because `(lo, hi)` + /// is itself a deterministic function of the input sample and the + /// percentiles, this satisfies "same sample + same percentiles ⇒ same + /// version": re-calibrating on a byte-identical sample with the same + /// percentile window reproduces the same `lo`, `hi`, and count, hence + /// the same hash. + /// + /// It does NOT guarantee the converse — two different samples that + /// happen to land on byte-identical `(lo, hi, count)` would share a + /// version — but that is correct rather than a gap: `(lo, hi)` is the + /// only state [`Self::quantize`] and [`Self::bucket_center`] ever + /// consult, so two floors with the same `(lo, hi)` really are the same + /// calibration epoch as far as decoding is concerned. + /// + /// `std::collections::hash_map::DefaultHasher` (SipHash) was + /// deliberately NOT used here: its seed is randomised per process, so + /// two runs of the identical calibration would produce two different + /// stamps — exactly the property this stamp must not have. + pub fn floor_version(&self) -> u64 { + self.floor_version + } + + /// Current `(lo, hi)` calibration bounds. + pub fn bounds(&self) -> (f64, f64) { + (self.lo, self.hi) + } + + /// Per-bucket occupancy counts over the calibration sample, length 256. + /// This **is** the histogram — no separate instrument is kept. See the + /// module docs. + pub fn occupancy(&self) -> &[u32; BUCKETS] { + &self.occupancy + } + + /// Number of finite values the calibration sample contributed to + /// [`Self::occupancy`] (non-finite values are excluded — see + /// [`calibrate`]). + pub fn samples(&self) -> u64 { + self.samples + } + + /// **COMPUTE — `&self`.** Fraction of the CALIBRATION SAMPLE landing in + /// the two rim buckets (`0` and `255`), read straight off + /// [`Self::occupancy`]. Returns `0.0` when [`Self::samples`] is `0`. + pub fn saturation(&self) -> f64 { + if self.samples == 0 { + return 0.0; + } + let rim = self.occupancy[0] as u64 + self.occupancy[BUCKETS - 1] as u64; + rim as f64 / self.samples as f64 + } + + /// **COMPUTE — `&self`.** Fraction of an ARBITRARY external population + /// `values` that lands in the two rim buckets under this floor. + /// + /// Unlike [`Self::saturation`] (which scores the calibration sample's + /// own recorded [`Self::occupancy`]), this re-quantises `values` + /// against this floor's `[lo, hi]` on the fly, without mutating + /// anything. Both directions bar B2 needs are "this floor scored + /// against a population it was not calibrated on": a narrow floor + /// scored globally, and a global floor scored on one narrow box. + /// Returns `0.0` on an empty `values` slice. + pub fn saturation_of(&self, values: &[f64]) -> f64 { + if values.is_empty() { + return 0.0; + } + let rim = values + .iter() + .filter(|&&v| { + let b = self.quantize(v); + b == 0 || b == (BUCKETS - 1) as u8 + }) + .count(); + rim as f64 / values.len() as f64 + } +} + +/// Deterministic FNV-1a 64-bit hash over raw bytes. See +/// [`CalibratedFloor::floor_version`] for why this, and not +/// `std::collections::hash_map::DefaultHasher`, backs the version stamp. +fn fnv1a_64(bytes: &[u8]) -> u64 { + const FNV_OFFSET_BASIS: u64 = 0xcbf2_9ce4_8422_2325; + const FNV_PRIME: u64 = 0x0000_0100_0000_01b3; + let mut hash = FNV_OFFSET_BASIS; + for &byte in bytes { + hash ^= byte as u64; + hash = hash.wrapping_mul(FNV_PRIME); + } + hash +} + +/// Linear-interpolated percentile (the "R-7" / `PERCENTILE.INC` method) +/// over an already-sorted, non-empty, all-finite slice. +/// +/// `p` is expected in `[0, 100]`. `rank = (p / 100) * (n - 1)`; +/// interpolates between the two adjacent order statistics when `rank` is +/// not an integer. +fn percentile(sorted: &[f64], p: f64) -> f64 { + let n = sorted.len(); + if n == 1 { + return sorted[0]; + } + let rank = (p / 100.0) * (n as f64 - 1.0); + let lo_idx = (rank.floor() as usize).min(n - 1); + let hi_idx = (rank.ceil() as usize).min(n - 1); + if lo_idx == hi_idx { + sorted[lo_idx] + } else { + let frac = rank - lo_idx as f64; + sorted[lo_idx] * (1.0 - frac) + sorted[hi_idx] * frac + } +} + +/// Calibrate a [`CalibratedFloor`] from a sample of `f64` values. +/// +/// Derives `[lo, hi]` at the [`LO_PERCENTILE`]–[`HI_PERCENTILE`] robust +/// window (see the module docs), then re-scans the sample under the +/// resulting floor to build [`CalibratedFloor::occupancy`] and +/// [`CalibratedFloor::samples`] — the "genuinely new" half of `D-WXS-3`; +/// [`CalibratedFloor::quantize`] / [`CalibratedFloor::bucket_center`] are +/// the re-expressed half. +/// +/// Non-finite (`NaN` / `±inf`) values in `sample` are excluded from both +/// the percentile computation and the occupancy scan — they carry no +/// ordering information a percentile window can use. +/// +/// Returns `None` when `sample` contains no finite values (there is +/// nothing to calibrate a window from). +pub fn calibrate(sample: &[f64]) -> Option { + let mut finite: Vec = sample.iter().copied().filter(|v| v.is_finite()).collect(); + if finite.is_empty() { + return None; + } + finite.sort_by(|a, b| a.total_cmp(b)); + + let lo = percentile(&finite, LO_PERCENTILE); + let hi = percentile(&finite, HI_PERCENTILE); + + let mut version_bytes: Vec = Vec::with_capacity(8 * 5); + version_bytes.extend_from_slice(&lo.to_le_bytes()); + version_bytes.extend_from_slice(&hi.to_le_bytes()); + version_bytes.extend_from_slice(&(finite.len() as u64).to_le_bytes()); + version_bytes.extend_from_slice(&LO_PERCENTILE.to_le_bytes()); + version_bytes.extend_from_slice(&HI_PERCENTILE.to_le_bytes()); + let floor_version = fnv1a_64(&version_bytes); + + let mut floor = CalibratedFloor { + lo, + hi, + floor_version, + occupancy: [0u32; BUCKETS], + samples: 0, + }; + + for &v in &finite { + let b = floor.quantize(v); + floor.occupancy[b as usize] = floor.occupancy[b as usize].saturating_add(1); + floor.samples = floor.samples.saturating_add(1); + } + + Some(floor) +} + +#[cfg(test)] +mod tests { + use super::*; + + // ── fixture helper ─────────────────────────────────────────────────── + + /// Evenly spaced points spanning `[lo, hi]` inclusive — deterministic, + /// no RNG, so every calibration and every measured saturation number in + /// these tests is exactly reproducible. + fn linspace(lo: f64, hi: f64, n: usize) -> Vec { + assert!(n >= 2, "linspace needs at least 2 points"); + (0..n) + .map(|i| lo + (hi - lo) * i as f64 / (n as f64 - 1.0)) + .collect() + } + + // ── B2 primary: well-covered sample, zero empty buckets, low saturation ─ + + #[test] + fn primary_well_covered_sample_has_zero_empty_buckets_and_low_saturation() { + // Pre-registered epsilon, worked out from the construction BEFORE + // running: the 0.4/99.6 percentile window trims ~0.4% of the + // 102_400-point evenly-spaced fixture off each end (~409-410 + // points per side), and each rim bucket (0 and 255) ALSO collects + // its own ~1/256 share of the interior mass (~397 points) because + // the trimmed points and the interior points nearest lo/hi both + // land in the same rim bucket. That puts expected rim occupancy + // near (409 + 397 + 410 + 397) / 102_400 ≈ 1.57%. epsilon = 0.03 + // leaves roughly 2x margin above that expectation without being so + // loose that a real quantisation bug (e.g. an inverted saturation + // branch, which would push saturation toward 1.0) would slip past. + const EPSILON: f64 = 0.03; + + let sample = linspace(-1000.0, 1000.0, 102_400); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + + let empty_buckets = floor.occupancy().iter().filter(|&&c| c == 0).count(); + assert_eq!( + empty_buckets, 0, + "expected every one of the 256 buckets occupied, found {empty_buckets} empty" + ); + + let sat = floor.saturation(); + assert!( + sat < EPSILON, + "saturation {sat} did not clear the pre-registered epsilon {EPSILON}" + ); + } + + // ── B2 control that can lose: narrow floor applied globally saturates ── + + #[test] + fn control_narrow_floor_applied_globally_shows_high_saturation() { + // This is `GEO-DEGENERATE` re-homed (plan §4 bar B2): a floor + // calibrated on a narrow slice, applied to a much wider + // independent population, must show HIGH saturation. Known + // losable — the arc measured 72-97% on real data for exactly this + // construction. + let narrow_sample = linspace(-10.0, 10.0, 2_000); + let narrow_floor = calibrate(&narrow_sample).expect("non-empty sample calibrates"); + + // Anti-vacuity: confirm the fixture really IS narrow before + // scoring it — otherwise a passing assertion below would prove + // nothing about the narrow-vs-wide mechanism. + let (n_lo, n_hi) = narrow_floor.bounds(); + assert!( + n_hi - n_lo < 25.0, + "fixture is not actually narrow: bounds [{n_lo}, {n_hi}]" + ); + + let wide_population = linspace(-1000.0, 1000.0, 102_400); + let sat = narrow_floor.saturation_of(&wide_population); + assert!( + sat > 0.9, + "expected high saturation applying a narrow floor globally, got {sat}" + ); + } + + // ── B2 stay-silent twin: global floor on a non-trivial sub-window ────── + + #[test] + fn stay_silent_global_floor_on_a_narrow_sub_window_does_not_saturate() { + // The global floor scored against a non-trivial sub-window of the + // SAME population it was calibrated on must NOT show high + // saturation. The sub-window here is the population's own upper + // third — real values, not an empty set — chosen specifically + // because it brushes against (without mostly exceeding) the + // calibration window's own trimmed edge, which is a genuinely + // testable case rather than a trivially-safe interior slice. + let wide_population = linspace(-1000.0, 1000.0, 102_400); + let global_floor = calibrate(&wide_population).expect("non-empty sample calibrates"); + + let n = wide_population.len(); + let sub_window = &wide_population[(2 * n / 3)..n]; + assert!( + sub_window.len() > 1_000, + "sub-window must be non-trivial, got {} points", + sub_window.len() + ); + + let sat = global_floor.saturation_of(sub_window); + assert!( + sat < 0.1, + "global floor should not saturate heavily on an interior-adjacent sub-window, got {sat}" + ); + } + + // ── round-trip bound ───────────────────────────────────────────────── + + #[test] + fn round_trip_bucket_center_is_within_half_bucket_of_source_value() { + let sample = linspace(-500.0, 1500.0, 50_000); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + let (lo, hi) = floor.bounds(); + assert!( + hi > lo, + "calibration must be non-degenerate for this fixture" + ); + + let half_bucket = (hi - lo) / (2.0 * BUCKETS as f64); + let probe_count = 5_000usize; + for i in 0..probe_count { + // Offset by half a probe-step so we never land exactly on a + // window endpoint, which would legitimately saturate. + let t = (i as f64 + 0.5) / probe_count as f64; + let v = lo + t * (hi - lo); + let b = floor.quantize(v); + let center = floor.bucket_center(b); + let delta = (center - v).abs(); + assert!( + delta <= half_bucket + 1e-9, + "round-trip error {delta} exceeds the ±half-bucket bound {half_bucket} at v={v}" + ); + } + } + + // ── version stamp is load-bearing ─────────────────────────────────── + + #[test] + fn version_stamp_mismatch_is_detected_and_matching_version_decodes() { + let sample = linspace(-50.0, 50.0, 5_000); + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + + let real_version = floor.floor_version(); + let bogus_version = real_version.wrapping_add(1); + assert_ne!( + bogus_version, real_version, + "fixture must construct a genuinely different version" + ); + + // Wrong version: must be detected, never silently mis-dequantized. + assert_eq!(floor.decode(10, bogus_version), None); + + // Matching version: must decode, to exactly what bucket_center gives. + let expected = floor.bucket_center(10); + assert_eq!(floor.decode(10, real_version), Some(expected)); + } + + // ── small direct-risk sanity checks on this file's own new code ──────── + + #[test] + fn calibrate_on_a_sample_with_no_finite_values_returns_none() { + let sample = [f64::NAN, f64::INFINITY, f64::NEG_INFINITY]; + assert!(calibrate(&sample).is_none()); + } + + #[test] + fn quantize_degenerate_bounds_returns_zero_matching_helix_guard() { + // An all-identical sample collapses the percentile window to + // lo == hi; `quantize` must not panic and must saturate every + // input to bucket 0 — the same degenerate-bounds guard + // `helix::quantize::RollingFloor::quantize` documents + // (`crates/helix/src/quantize.rs:100-102`). + let sample = vec![42.0_f64; 500]; + let floor = calibrate(&sample).expect("non-empty sample calibrates"); + let (lo, hi) = floor.bounds(); + assert!( + hi <= lo, + "an all-identical sample should collapse the window, got [{lo}, {hi}]" + ); + assert_eq!(floor.quantize(0.0), 0); + assert_eq!(floor.quantize(42.0), 0); + assert_eq!(floor.quantize(1000.0), 0); + } +} diff --git a/crates/weather-poc/src/key.rs b/crates/weather-poc/src/key.rs new file mode 100644 index 000000000..453cde51a --- /dev/null +++ b/crates/weather-poc/src/key.rs @@ -0,0 +1,348 @@ +//! Grid cell (lat_idx, lon_idx) <-> the canonical 16-byte `NodeGuid` key. +//! +//! Owner: the key worker (plan `.claude/plans/weather-soa-bake-v1.md` §6.2, +//! deliverable D-WXS-2). Grid: ERA5 0.25 degree, pole-inclusive latitude +//! (`lat_idx` 0..721), cyclic longitude (`lon_idx` 0..1440). +//! +//! # Byte layout +//! +//! This crate is deliberately zero-dep (`Cargo.toml`'s "ZERO-DEP BY +//! CONSTRUCTION" note) and therefore has no dependency on +//! `lance-graph-contract`'s `NodeGuid` type. [`encode_key`] instead emits a +//! raw `[u8; 16]` in the **same byte layout** `NodeGuid` uses (`CLAUDE.md` +//! § "CANON -- Minimal SoA node"), little-endian throughout: +//! +//! | bytes | content | +//! |---|---| +//! | `0..4` | `classid: u32`, little-endian -- a **parameter this module accepts**; it is never composed, minted, or bit-manipulated here (`D-WXS-0`, the classid mint, is blocked on an OGAR-side decision -- plan §1.4) | +//! | `4` | HEEL, lat axis: `(lat_idx >> 6) as u8` | +//! | `5` | HEEL, lon axis: `(lon_idx >> 6) as u8` | +//! | `6` | HIP, lat axis: `(lat_idx & 63) as u8` | +//! | `7` | HIP, lon axis: `(lon_idx & 63) as u8` | +//! | `8..10` | TWIG -- `0u16`, dormant/reserved (zero-fallback: "not consulted", never "compacted") | +//! | `10..16`| untouched (all zero) -- the V1-legacy tail; this crate mints nothing there | +//! +//! **Agreement between this layout and the real `NodeGuid` (`canonical_node.rs`) +//! is NOT verified by this module.** That is a separate, cross-manifest parity +//! deliverable of the same shape as `D-WXS-12` (jc <-> ndarray) -- a measured +//! comparison run across both crates, not a dependency edge from this +//! (deliberately dependency-free) crate. +//! +//! # The three wrinkles this module implements (plan §1.3) +//! +//! 1. **Ragged tiles, never padded.** `721 = 11*64 + 17` and +//! `1440 / 64 = 22.5`: the last tile on each axis is partial (17 lat rows, +//! 32 lon columns). Cells are addressed by index with half-open ranges; +//! nothing is padded to a full tile, because a padded cell would be a +//! cell that does not exist. +//! 2. **Longitude is cyclic, latitude is not.** A box that crosses the +//! 0deg/360deg seam is decomposed into two ranges by [`box_ranges`], never +//! approximated as one. Latitude has no such wrap -- it has poles instead +//! (a pole row is one physical point repeated `LON_COUNT` times, which is +//! a downstream statistics concern, not a key-encoding one). +//! 3. **No `MailboxId`/`NiblePath` here.** That mapping is unresolved +//! elsewhere in the workspace (`le-contract.md` §5) and this module does +//! not lean on it. + +/// Number of latitude rows in the ERA5 0.25 degree, pole-inclusive grid +/// (`720 + 1`). `lat_idx` ranges over `0..LAT_COUNT`. +/// +/// `721 = 11 * 64 + 17`: the grid does not tile evenly at the 64-row tile +/// size, so the last latitude tile (HEEL = 11) is ragged, holding only the +/// 17 rows `704..721` (plan §1.3 wrinkle 1). +pub const LAT_COUNT: u16 = 721; + +/// Number of longitude columns in the ERA5 0.25 degree grid. `lon_idx` +/// ranges over `0..LON_COUNT` and is cyclic (wraps modulo `LON_COUNT`). +/// +/// `1440 / 64 = 22.5`: the last longitude tile (HEEL = 22) is ragged, +/// holding only the 32 columns `1408..1440` (plan §1.3 wrinkle 1). +pub const LON_COUNT: u16 = 1440; + +/// Length in bytes of the canonical `NodeGuid` key this module emits. +pub const KEY_LEN: usize = 16; + +/// The HEEL/HIP split point on each axis: `HEEL = idx >> TILE_SHIFT`, +/// `HIP = idx & TILE_MASK`. A 64-wide tile per axis. +const TILE_SHIFT: u16 = 6; + +/// `(1 << TILE_SHIFT) - 1` -- the low-bits mask recovering the HIP (within +/// tile) component of a grid index. +const TILE_MASK: u16 = (1 << TILE_SHIFT) - 1; + +/// Encodes a grid cell `(lat_idx, lon_idx)` into the canonical 16-byte key, +/// under the caller-supplied `classid`. +/// +/// `classid` is stored verbatim as little-endian bytes -- this module never +/// composes, mints, or bit-manipulates it (`D-WXS-0` is blocked; see the +/// module doc's byte-layout table). +/// +/// # Preconditions +/// +/// `lat_idx < LAT_COUNT` and `lon_idx < LON_COUNT`. Violating this is a +/// caller bug: in debug builds it trips a `debug_assert`; in release builds +/// the HEEL/HIP split still runs (silently producing a key that will not +/// round-trip through [`decode_key`], since the out-of-range guard lives on +/// the decode side, not here). Callers that need to construct or inspect an +/// out-of-range key directly (e.g. to test rejection) should build the +/// `[u8; 16]` by hand rather than calling this function with an +/// out-of-range index. +pub fn encode_key(classid: u32, lat_idx: u16, lon_idx: u16) -> [u8; KEY_LEN] { + debug_assert!( + lat_idx < LAT_COUNT, + "lat_idx {lat_idx} out of range (0..{LAT_COUNT})" + ); + debug_assert!( + lon_idx < LON_COUNT, + "lon_idx {lon_idx} out of range (0..{LON_COUNT})" + ); + + let mut key = [0u8; KEY_LEN]; + key[0..4].copy_from_slice(&classid.to_le_bytes()); + key[4] = (lat_idx >> TILE_SHIFT) as u8; + key[5] = (lon_idx >> TILE_SHIFT) as u8; + key[6] = (lat_idx & TILE_MASK) as u8; + key[7] = (lon_idx & TILE_MASK) as u8; + // bytes 8..10 (TWIG) stay 0 -- dormant, reserved. + // bytes 10..16 stay 0 -- V1-legacy tail; this crate mints nothing there. + key +} + +/// Decodes a canonical 16-byte key back into its grid cell `(lat_idx, +/// lon_idx)`, or `None` if the encoded cell falls outside the real grid +/// (`0..LAT_COUNT` x `0..LON_COUNT`). +/// +/// The bound check is deliberately on the *reconstructed* index +/// (`heel << TILE_SHIFT | hip`), not on the HEEL/HIP bytes individually -- +/// this is what makes an out-of-range cell (e.g. `lat_idx == LAT_COUNT`, +/// one past the last valid row) come back `None` instead of silently +/// wrapping or aliasing onto a real cell. `classid` and the TWIG/tail bytes +/// are not inspected; only the lat/lon axes are decoded. +pub fn decode_key(key: &[u8; KEY_LEN]) -> Option<(u16, u16)> { + let heel_lat = key[4] as u16; + let heel_lon = key[5] as u16; + let hip_lat = key[6] as u16; + let hip_lon = key[7] as u16; + + let lat_idx = (heel_lat << TILE_SHIFT) | hip_lat; + let lon_idx = (heel_lon << TILE_SHIFT) | hip_lon; + + if lat_idx >= LAT_COUNT || lon_idx >= LON_COUNT { + return None; + } + Some((lat_idx, lon_idx)) +} + +/// A rectangular, half-open, non-wrapping box of grid cells: +/// `lat_lo..lat_hi` and `lon_lo..lon_hi`. +/// +/// The only place a longitude wrap is resolved is [`box_ranges`] -- every +/// `CellBox` it hands back already excludes the wrap; a `CellBox` value is +/// never itself wrapping. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct CellBox { + /// Inclusive lower latitude index. + pub lat_lo: u16, + /// Exclusive upper latitude index. + pub lat_hi: u16, + /// Inclusive lower longitude index. + pub lon_lo: u16, + /// Exclusive upper longitude index. + pub lon_hi: u16, +} + +/// Splits a lat/lon box into the non-wrapping [`CellBox`] range(s) that +/// cover it (plan §1.3 wrinkle 2: "a box crossing the 0deg/360deg seam is +/// two ranges, never one"). +/// +/// `lat_lo..lat_hi` is passed through unchanged on every returned box -- +/// latitude never wraps (it has poles, not a seam). +/// +/// `lon_lo` and `lon_hi` are both grid indices in `0..=LON_COUNT`: +/// +/// * If `lon_lo < lon_hi`, the box does not cross the seam and this returns +/// **exactly one** `CellBox` covering `lon_lo..lon_hi`. +/// * If `lon_lo >= lon_hi`, the box wraps (the caller is expressing "from +/// `lon_lo` around through 0deg to `lon_hi`") and this returns **exactly +/// two**: `lon_lo..LON_COUNT`, then `0..lon_hi`. +/// +/// The return type is deliberate: a `Vec` (never an `Option`) so a +/// caller cannot silently drop the second range the way an optional-single- +/// range API would let them -- callers must handle however many ranges come +/// back, and `.len()` itself reports whether the box wrapped. +pub fn box_ranges(lat_lo: u16, lat_hi: u16, lon_lo: u16, lon_hi: u16) -> Vec { + if lon_lo < lon_hi { + vec![CellBox { + lat_lo, + lat_hi, + lon_lo, + lon_hi, + }] + } else { + vec![ + CellBox { + lat_lo, + lat_hi, + lon_lo, + lon_hi: LON_COUNT, + }, + CellBox { + lat_lo, + lat_hi, + lon_lo: 0, + lon_hi, + }, + ] + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashSet; + + /// Primary bar (B1): every one of the 1,038,240 real ERA5 0.25-degree + /// grid cells (`LAT_COUNT * LON_COUNT`) round-trips exactly through + /// `encode_key` -> `decode_key`, AND no two distinct cells produce the + /// same 16 bytes. This is **exhaustive, not sampled** -- ~1M iterations + /// is fast and this is the primary falsifier for the whole module: any + /// bug that makes two cells collide, or makes any single cell fail to + /// round-trip, fails this test. + #[test] + fn all_grid_cells_round_trip_and_are_collision_free() { + let classid = 0x1234_5678u32; + let total = LAT_COUNT as usize * LON_COUNT as usize; + assert_eq!(total, 1_038_240, "grid cell count drifted from the plan"); + + let mut seen: HashSet<[u8; KEY_LEN]> = HashSet::with_capacity(total); + for lat_idx in 0..LAT_COUNT { + for lon_idx in 0..LON_COUNT { + let key = encode_key(classid, lat_idx, lon_idx); + assert_eq!( + decode_key(&key), + Some((lat_idx, lon_idx)), + "round-trip failed for ({lat_idx}, {lon_idx})" + ); + assert!( + seen.insert(key), + "collision: ({lat_idx}, {lon_idx}) produced a key already seen" + ); + } + } + assert_eq!(seen.len(), total, "fewer distinct keys than grid cells"); + } + + /// Control that can lose: exercises the ragged last tile on BOTH axes + /// explicitly, using hardcoded bounds (704/721, 1408/1440) rather than + /// `LAT_COUNT`/`LON_COUNT`. If the exhaustive sweep above ever had its + /// bounds accidentally clamped down to the last *full* tile (704 lat + /// rows, 1408 lon columns -- e.g. a copy-pasted `& !TILE_MASK`), this + /// test would still catch it, because it does not share that bug's + /// source of truth. + #[test] + fn ragged_last_tile_cells_round_trip_on_both_axes() { + let classid = 0x0102_0304u32; + for lat_idx in 704..721u16 { + for lon_idx in 1408..1440u16 { + let key = encode_key(classid, lat_idx, lon_idx); + assert_eq!( + decode_key(&key), + Some((lat_idx, lon_idx)), + "ragged-tile cell ({lat_idx}, {lon_idx}) did not round-trip" + ); + } + } + } + + /// Stay-silent-twin partner of the ragged-tile test above, on the + /// rejection side: a cell exactly one past the grid on each axis must + /// be rejected by `decode_key`, not silently wrapped or aliased onto a + /// real cell. Keys are built by hand (not via `encode_key`, whose + /// precondition these indices violate) so the test exercises + /// `decode_key`'s own bound check in isolation. + #[test] + fn out_of_range_index_is_rejected_not_silently_wrapped() { + let classid = 0xAABB_CCDDu32; + + // lat_idx = 721 = 11*64 + 17: heel=11 (valid tile), hip=17 (one + // past that tile's 17 valid rows, 0..=16). + let mut lat_over = [0u8; KEY_LEN]; + lat_over[0..4].copy_from_slice(&classid.to_le_bytes()); + lat_over[4] = 11; // lat heel + lat_over[5] = 0; // lon heel (valid) + lat_over[6] = 17; // lat hip -> reconstructs to 721 + lat_over[7] = 0; // lon hip + assert_eq!( + decode_key(&lat_over), + None, + "lat_idx=721 (one past LAT_COUNT) must be rejected" + ); + + // lon_idx = 1440 = 22*64 + 32: heel=22 (valid tile), hip=32 (one + // past that tile's 32 valid columns, 0..=31). + let mut lon_over = [0u8; KEY_LEN]; + lon_over[0..4].copy_from_slice(&classid.to_le_bytes()); + lon_over[4] = 0; // lat heel (valid) + lon_over[5] = 22; // lon heel + lon_over[6] = 0; // lat hip + lon_over[7] = 32; // lon hip -> reconstructs to 1440 + assert_eq!( + decode_key(&lon_over), + None, + "lon_idx=1440 (one past LON_COUNT) must be rejected" + ); + + // Positive control: the true last valid cell on both axes must + // still decode fine -- proving the two rejections above are about + // being genuinely out of range, not a blanket "anything near the + // boundary fails" bug. + let last_valid = encode_key(classid, 720, 1439); + assert_eq!(decode_key(&last_valid), Some((720, 1439))); + } + + /// `box_ranges` can fire: a box crossing the 0deg/360deg seam reports + /// the wrap as exactly two ranges. + #[test] + fn box_ranges_seam_crossing_reports_exactly_two_ranges() { + // lon 1400..1440, wrapping around to lon 0..40. + let ranges = box_ranges(10, 20, 1400, 40); + assert_eq!(ranges.len(), 2, "seam-crossing box must report the wrap"); + assert_eq!( + ranges[0], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 1400, + lon_hi: LON_COUNT, + } + ); + assert_eq!( + ranges[1], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 0, + lon_hi: 40, + } + ); + } + + /// `box_ranges` can stay silent: an ordinary box entirely inside + /// `0..LON_COUNT` reports no wrap -- exactly one range, matching the + /// input bounds verbatim. Paired with the can-fire test above so + /// neither half of the wrap detector goes unchecked. + #[test] + fn box_ranges_non_wrapping_box_reports_exactly_one_range() { + let ranges = box_ranges(10, 20, 200, 400); + assert_eq!(ranges.len(), 1, "non-wrapping box must report no wrap"); + assert_eq!( + ranges[0], + CellBox { + lat_lo: 10, + lat_hi: 20, + lon_lo: 200, + lon_hi: 400, + } + ); + } +} diff --git a/crates/weather-poc/src/lib.rs b/crates/weather-poc/src/lib.rs new file mode 100644 index 000000000..40d5801e8 --- /dev/null +++ b/crates/weather-poc/src/lib.rs @@ -0,0 +1,41 @@ +//! Zarr → `NodeRow`: the ERA5 0.25° global grid on the canonical SoA row. +//! +//! **Plan:** `.claude/plans/weather-soa-bake-v1.md`. This crate exists because +//! the weather R&D arc ran entirely as Python over a Zarr file — one variable, +//! four hand-picked 16° boxes, three hand-typed timesteps — and the reason was +//! structural: the Zarr→`NodeRow` path did not exist and no plan specified one. +//! +//! # The shape, in four facts +//! +//! * **One cell is one node.** The ERA5 0.25° grid is 721 × 1440 = +//! **1,038,240 cells**; the canonical row is 512 bytes; so a full global +//! timestep is **0.495 GiB**. All 122 ERA5 fields per cell (17 surface + 7 +//! pressure-level × 13 levels + 14 static) fit at 1 byte/field inside the +//! **292 free slab bytes** — see [`crate::key`] and the plan §2.1 for why the +//! budget is 292 and not 384. +//! * **The key is the address.** `HEEL` = the 16° × 16° tile, `HIP` = position +//! within it, `TWIG` dormant-reserved. A lat/lon grid is the literal-x/y case +//! of the 3×4 cascade, so the arc's hand-picked 16° boxes become **HEEL-prefix +//! range scans** rather than array slicing. +//! * **One timestep is one Lance version.** Per `E-MARKOV-TEMPORAL-STREAM-1`, +//! episodic = Lance versions; a time series is a version-range read. This +//! crate never writes a version writer of its own. +//! * **Nothing in the payload says what a byte means.** The (facet, pair, byte) +//! → (variable, level, unit, floor) mapping is a ClassView-side manifest +//! ([`crate::manifest`]), never a slot in the row — the le-contract §2 +//! slot-purity rule. +//! +//! # Status +//! +//! Wave W1 is under construction. `D-WXS-0` (the classid mint) is **blocked** +//! on an OGAR-side, operator-gated decision, and until it resolves the bake +//! must **refuse to write** rather than emit rows under `0x0000_0000` — that +//! value belongs to the zero-fallback ladder and a dataset carrying it is +//! indistinguishable from a bootstrap row. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +pub mod floor; +pub mod key; +pub mod manifest; diff --git a/crates/weather-poc/src/manifest.rs b/crates/weather-poc/src/manifest.rs new file mode 100644 index 000000000..92db70cff --- /dev/null +++ b/crates/weather-poc/src/manifest.rs @@ -0,0 +1,615 @@ +//! The ClassView-side field manifest: `(facet, pair, byte) <-> (ERA5 +//! variable, level, unit, floor id)`. +//! +//! # Why this exists (le-contract.md §2, slot purity) +//! +//! **"Labels and positions come from the ClassView, NEVER from a slot in the +//! payload."** The `NodeRow`'s value slab is dumb bytes; a byte carries no +//! variable name, no level number, no unit, no display label. The mapping +//! *"F1 pair 0 hi-byte = `v_component_of_wind` at 850 hPa, unit m/s, floor id +//! `v850`"* cannot live in the row — it lives here, as a committed data +//! artifact the bake reads. This module is ClassView-side metadata; nothing +//! it describes is ever written into a `NodeRow`. +//! +//! # The lane (plan `.claude/plans/weather-soa-bake-v1.md` §2.2) +//! +//! L4, `6 x (8:8)` palette256² (`.claude/v3/soa_layout/le-contract.md` §3): +//! each 16-byte facet is a 4-byte classid prefix followed by six `(lo, hi)` +//! byte pairs. `u8:u8` means two SEPARATE bytes — never widened to u16. Each +//! byte carries one ERA5 field quantised to 256 levels; a pair holds a +//! physically paired couple (e.g. wind components), so the pair's own +//! similarity is one 256×256 table read. +//! +//! # The committed W1 field set (plan §2.5) +//! +//! Three facets, F0 (surface), F1 (850 hPa), F2 (500 hPa); five occupied +//! pairs in F0 and three each in F1/F2. Every other pair is reserved-zero — +//! **absent from this manifest entirely**, per the "reserved slots are NOT +//! rows" rule: a slot with no entry is dormant by construction, expandable +//! later with zero layout change (RESERVE, DON'T RECLAIM). +//! +//! # `lo`/`hi` convention +//! +//! Within a pair written `A : B` in the plan's table, `A` is the LOW byte +//! and `B` is the HIGH byte — the pair's own left-to-right order, kept +//! consistent across the whole committed manifest (see the comment header +//! of `data/field_manifest_v1.tsv`). +//! +//! # Bar B0 — what is implemented here, and what is deferred +//! +//! Plan §4 W0's bar B0 is *"mutating one manifest entry must change the +//! bytes the bake writes for at least one cell."* **The bake does not exist +//! yet** (`D-WXS-4`, a separate, later deliverable), so that end-to-end half +//! cannot be exercised from this module. What IS implemented and tested +//! here is the manifest-level half of B0: resolution is real (a lookup +//! finds the right slot and nothing else), and a manifest whose two entries +//! collide on one `(facet, pair, byte)` slot is rejected outright. The +//! stay-silent twin — a manifest that differs byte-for-byte (reordered +//! rows) but is semantically identical still resolves every lookup +//! identically — is also tested. Wiring bar B0's end-to-end half is `D-WXS-4`'s +//! job, not this module's. + +use std::fmt; + +/// The exact header line every field-manifest TSV must start with (after any +/// leading `#` comment / blank lines, which are skipped). +const HEADER: &str = "facet\tpair\tbyte\tvariable\tlevel_hpa\tunit\tfloor_id"; + +/// The committed W1 field manifest, embedded at compile time so this crate +/// stays dependency-free (no TSV/CSV crate, no serde — plan §6.3). +const COMMITTED_MANIFEST_TSV: &str = include_str!("../data/field_manifest_v1.tsv"); + +/// Which byte of an `(8:8)` pair a manifest entry occupies. +/// +/// `u8:u8` means two separate bytes, never a widened `u16` — le-contract.md +/// §1/§3. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PairByte { + /// The low byte of the pair (the first-listed variable in the plan's + /// `A : B` table entries). + Lo, + /// The high byte of the pair (the second-listed variable). + Hi, +} + +impl fmt::Display for PairByte { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + PairByte::Lo => f.write_str("lo"), + PairByte::Hi => f.write_str("hi"), + } + } +} + +/// One row of the field manifest: the (facet, pair, byte) slot a single +/// ERA5 field occupies, plus its physical unit and calibration floor id. +/// +/// Nothing on this type is ever written into a `NodeRow` — it is +/// ClassView-side metadata the bake and any reader consults, per the +/// module-level slot-purity note. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ManifestEntry { + /// Facet index: `0` = F0 (surface), `1` = F1 (850 hPa), `2` = F2 + /// (500 hPa). Always `0..=2` for a validated manifest. + pub facet: u8, + /// Pair index within the facet, `0..=5`. + pub pair: u8, + /// Which byte of the pair this entry occupies. + pub byte: PairByte, + /// The ERA5 variable name, exactly as it appears in the Zarr store + /// (e.g. `u_component_of_wind`, `mean_sea_level_pressure`). + pub variable: String, + /// The pressure level in hPa, or `None` for a surface field (the TSV's + /// literal `surface` token). + pub level_hpa: Option, + /// The physical unit (e.g. `Pa`, `K`, `m s-1`, `kg kg-1`, `m2 s-2`, `1` + /// for a dimensionless fraction). + pub unit: String, + /// A stable identifier for the calibration floor this field uses. + /// Two entries sharing a `floor_id` share the same `[lo, hi]` + /// calibration (plan §2.3/D-WXS-3); none do in the committed W1 + /// manifest today. + pub floor_id: String, +} + +/// Everything that can go wrong parsing or validating field-manifest text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ManifestError { + /// The first non-comment, non-blank line did not match [`HEADER`] + /// exactly, or no such line existed at all (`line == 0`). + BadHeader { + /// 1-based source line number (`0` if no header line was found at + /// all). + line: usize, + /// The line actually found, if any. + found: String, + }, + /// A data row did not split into exactly 7 tab-separated columns. + WrongColumnCount { + /// 1-based source line number. + line: usize, + /// How many columns were actually found. + found: usize, + }, + /// The `facet` column was not `0`, `1`, or `2`. + FacetOutOfRange { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `pair` column was not in `0..=5`. + PairOutOfRange { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `byte` column was neither `lo` nor `hi`. + BadByteToken { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// The `level_hpa` column was neither the literal `surface` nor a + /// parseable non-negative integer. + BadLevel { + /// 1-based source line number. + line: usize, + /// The raw column text. + value: String, + }, + /// Two entries claim the same `(facet, pair, byte)` slot — the manifest + /// is not load-bearing if this is possible (bar B0). + SlotCollision { + /// The colliding facet index. + facet: u8, + /// The colliding pair index. + pair: u8, + /// The colliding byte. + byte: PairByte, + /// The variable name of the first entry claiming the slot. + first_variable: String, + /// The variable name of the second entry claiming the slot. + second_variable: String, + }, +} + +impl fmt::Display for ManifestError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + ManifestError::BadHeader { line, found } => write!( + f, + "manifest line {line}: expected header {HEADER:?}, found {found:?}" + ), + ManifestError::WrongColumnCount { line, found } => write!( + f, + "manifest line {line}: expected 7 tab-separated columns, found {found}" + ), + ManifestError::FacetOutOfRange { line, value } => { + write!(f, "manifest line {line}: facet {value:?} is not 0, 1, or 2") + } + ManifestError::PairOutOfRange { line, value } => { + write!(f, "manifest line {line}: pair {value:?} is not in 0..=5") + } + ManifestError::BadByteToken { line, value } => write!( + f, + "manifest line {line}: byte token {value:?} is neither \"lo\" nor \"hi\"" + ), + ManifestError::BadLevel { line, value } => write!( + f, + "manifest line {line}: level_hpa {value:?} is neither \"surface\" nor an integer" + ), + ManifestError::SlotCollision { + facet, + pair, + byte, + first_variable, + second_variable, + } => write!( + f, + "manifest slot (facet={facet}, pair={pair}, byte={byte}) is claimed by both {first_variable:?} and {second_variable:?}" + ), + } + } +} + +impl std::error::Error for ManifestError {} + +/// A parsed, validated field manifest: an ordered list of [`ManifestEntry`] +/// rows plus the two resolution directions over them. +#[derive(Debug, Clone)] +pub struct FieldManifest { + entries: Vec, +} + +impl FieldManifest { + /// Parses and validates field-manifest text. + /// + /// Lines that are blank or start with `#` are skipped wherever they + /// occur. The first remaining line must equal [`HEADER`] exactly (a + /// permissive "starts with" check would be a silent-misread waiting + /// for a schema change). Every remaining line is a data row of exactly + /// 7 tab-separated columns. After every row parses, the full set is + /// validated for `(facet, pair, byte)` collisions. + /// + /// # Errors + /// + /// Returns the first [`ManifestError`] encountered — a malformed + /// header, a malformed row, or (only after every row has parsed + /// successfully) a slot collision between two otherwise-valid rows. + pub fn parse(text: &str) -> Result { + let mut header_seen = false; + let mut entries = Vec::new(); + + for (line_no, raw_line) in text.lines().enumerate().map(|(i, l)| (i + 1, l)) { + let line = raw_line.trim_end(); + if line.is_empty() || line.starts_with('#') { + continue; + } + + if !header_seen { + if line != HEADER { + return Err(ManifestError::BadHeader { + line: line_no, + found: line.to_string(), + }); + } + header_seen = true; + continue; + } + + let cols: Vec<&str> = line.split('\t').collect(); + if cols.len() != 7 { + return Err(ManifestError::WrongColumnCount { + line: line_no, + found: cols.len(), + }); + } + + let facet: u8 = cols[0] + .parse::() + .ok() + .filter(|v| *v <= 2) + .ok_or_else(|| ManifestError::FacetOutOfRange { + line: line_no, + value: cols[0].to_string(), + })?; + + let pair: u8 = cols[1] + .parse::() + .ok() + .filter(|v| *v <= 5) + .ok_or_else(|| ManifestError::PairOutOfRange { + line: line_no, + value: cols[1].to_string(), + })?; + + let byte = match cols[2] { + "lo" => PairByte::Lo, + "hi" => PairByte::Hi, + other => { + return Err(ManifestError::BadByteToken { + line: line_no, + value: other.to_string(), + }) + } + }; + + let variable = cols[3].to_string(); + + let level_hpa = if cols[4] == "surface" { + None + } else { + Some( + cols[4] + .parse::() + .map_err(|_| ManifestError::BadLevel { + line: line_no, + value: cols[4].to_string(), + })?, + ) + }; + + let unit = cols[5].to_string(); + let floor_id = cols[6].to_string(); + + entries.push(ManifestEntry { + facet, + pair, + byte, + variable, + level_hpa, + unit, + floor_id, + }); + } + + if !header_seen { + return Err(ManifestError::BadHeader { + line: 0, + found: String::new(), + }); + } + + validate_no_slot_collisions(&entries)?; + + Ok(FieldManifest { entries }) + } + + /// The committed W1 field manifest (`data/field_manifest_v1.tsv`). + /// + /// # Panics + /// + /// Panics if the committed artifact fails to parse or validate. That is + /// a build-time invariant, not a runtime one — the committed TSV is + /// meant to always be well-formed, and a session breaking it should see + /// the failure immediately rather than downstream. Use [`Self::parse`] + /// directly for a fallible read of arbitrary manifest text. + pub fn committed() -> Self { + Self::parse(COMMITTED_MANIFEST_TSV) + .expect("committed data/field_manifest_v1.tsv must parse and validate") + } + + /// Every entry in the manifest, in file order. + pub fn entries(&self) -> &[ManifestEntry] { + &self.entries + } + + /// Resolves `(variable, level_hpa)` to the `(facet, pair, byte)` slot it + /// occupies, or `None` if this manifest has no entry for it (e.g. a + /// pressure-level variable at a level outside the W1 set). + pub fn resolve(&self, variable: &str, level_hpa: Option) -> Option<(u8, u8, PairByte)> { + self.entries + .iter() + .find(|e| e.variable == variable && e.level_hpa == level_hpa) + .map(|e| (e.facet, e.pair, e.byte)) + } + + /// The inverse of [`Self::resolve`]: which entry (if any) occupies a + /// given `(facet, pair, byte)` slot. + pub fn resolve_slot(&self, facet: u8, pair: u8, byte: PairByte) -> Option<&ManifestEntry> { + self.entries + .iter() + .find(|e| e.facet == facet && e.pair == pair && e.byte == byte) + } +} + +/// Rejects a manifest where two entries claim the same `(facet, pair, +/// byte)` slot. `O(n^2)` over the entry count, which is fine at the W1 +/// scale (22 entries) and simple enough to audit by inspection. +fn validate_no_slot_collisions(entries: &[ManifestEntry]) -> Result<(), ManifestError> { + for i in 0..entries.len() { + for j in (i + 1)..entries.len() { + let a = &entries[i]; + let b = &entries[j]; + if a.facet == b.facet && a.pair == b.pair && a.byte == b.byte { + return Err(ManifestError::SlotCollision { + facet: a.facet, + pair: a.pair, + byte: a.byte, + first_variable: a.variable.clone(), + second_variable: b.variable.clone(), + }); + } + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Parses and validates: the committed TSV loads, every entry is in + /// range, and the per-facet slot count matches the plan §2.5 table + /// exactly (`==`, not `>=` — a permissive arity guard is a silent + /// misread waiting for a schema change). + #[test] + fn committed_manifest_parses_and_every_entry_is_in_range() { + let manifest = FieldManifest::committed(); + + assert_eq!(manifest.entries().len(), 22); + + for entry in manifest.entries() { + assert!(entry.facet <= 2, "facet {} out of range", entry.facet); + assert!(entry.pair <= 5, "pair {} out of range", entry.pair); + assert!(!entry.variable.is_empty()); + assert!(!entry.unit.is_empty()); + assert!(!entry.floor_id.is_empty()); + } + + let count = |facet: u8| { + manifest + .entries() + .iter() + .filter(|e| e.facet == facet) + .count() + }; + assert_eq!(count(0), 10, "F0 (surface): 5 occupied pairs x 2 bytes"); + assert_eq!(count(1), 6, "F1 (850 hPa): 3 occupied pairs x 2 bytes"); + assert_eq!(count(2), 6, "F2 (500 hPa): 3 occupied pairs x 2 bytes"); + } + + /// Can-it-fire: a manifest with two entries claiming the same + /// `(facet, pair, byte)` slot must be rejected. + #[test] + fn colliding_entries_are_rejected() { + let text = format!( + "{HEADER}\n0\t0\tlo\tmean_sea_level_pressure\tsurface\tPa\tmsl\n0\t0\tlo\tsurface_pressure\tsurface\tPa\tsp\n" + ); + + let err = FieldManifest::parse(&text).expect_err("colliding slots must be rejected"); + match err { + ManifestError::SlotCollision { + facet, pair, byte, .. + } => { + assert_eq!(facet, 0); + assert_eq!(pair, 0); + assert_eq!(byte, PairByte::Lo); + } + other => panic!("expected SlotCollision, got {other:?}"), + } + } + + /// Stay-silent twin of the collision guard: two entries on DIFFERENT + /// bytes of the same pair must be accepted, not flagged as a collision. + /// A validator that rejects everything carries exactly as much + /// information as one that accepts everything. + #[test] + fn non_colliding_entries_on_the_same_pair_are_accepted() { + let text = format!( + "{HEADER}\n0\t0\tlo\tmean_sea_level_pressure\tsurface\tPa\tmsl\n0\t0\thi\tsurface_pressure\tsurface\tPa\tsp\n" + ); + + let manifest = FieldManifest::parse(&text).expect("non-colliding rows must parse"); + assert_eq!(manifest.entries().len(), 2); + } + + /// Stay-silent twin (non-trivial, over the real committed data): a + /// manifest that is byte-different — the data rows in reverse order — + /// but semantically identical to the committed one must be accepted + /// AND must resolve every lookup identically. Mutating a reserved-zero + /// slot's description or reordering rows must change not one resolved + /// answer. + #[test] + fn reordering_the_committed_rows_changes_no_resolution() { + let lines: Vec<&str> = COMMITTED_MANIFEST_TSV.lines().collect(); + let header_idx = lines + .iter() + .position(|l| !l.trim().is_empty() && !l.starts_with('#')) + .expect("committed manifest must have a header line"); + let (head, data) = lines.split_at(header_idx + 1); + let mut reordered_data: Vec<&str> = data.to_vec(); + reordered_data.reverse(); + + let mut reordered_text = head.join("\n"); + reordered_text.push('\n'); + reordered_text.push_str(&reordered_data.join("\n")); + reordered_text.push('\n'); + + // Sanity: the text really is byte-different from the original. + assert_ne!(reordered_text, COMMITTED_MANIFEST_TSV); + + let original = FieldManifest::committed(); + let reordered = + FieldManifest::parse(&reordered_text).expect("reordered manifest must still validate"); + + assert_eq!(original.entries().len(), reordered.entries().len()); + for entry in original.entries() { + assert_eq!( + original.resolve(&entry.variable, entry.level_hpa), + reordered.resolve(&entry.variable, entry.level_hpa), + "resolution for {:?} at {:?} diverged after reordering", + entry.variable, + entry.level_hpa + ); + } + } + + /// Resolution is real: a known `(variable, level)` resolves to its + /// documented slot, and the inverse lookup round-trips back to the same + /// entry. + #[test] + fn resolves_a_known_slot_and_its_inverse_round_trips() { + let manifest = FieldManifest::committed(); + + let (facet, pair, byte) = manifest + .resolve("u_component_of_wind", Some(850)) + .expect("u_component_of_wind@850 must resolve"); + assert_eq!((facet, pair), (1, 0)); + assert_eq!(byte, PairByte::Lo); + + let entry = manifest + .resolve_slot(facet, pair, byte) + .expect("the inverse lookup must find the entry resolve() just returned"); + assert_eq!(entry.variable, "u_component_of_wind"); + assert_eq!(entry.level_hpa, Some(850)); + assert_eq!(entry.floor_id, "u850"); + } + + /// The other half of resolution being real: a `(variable, level)` pair + /// outside the W1 set resolves to `None`, not to a wrong slot. + #[test] + fn a_variable_level_outside_the_w1_set_does_not_resolve() { + let manifest = FieldManifest::committed(); + // temperature exists at 850/500 in W1, but not at 250. + assert_eq!(manifest.resolve("temperature", Some(250)), None); + // geopotential exists as a pressure-level field, never as a surface one. + assert_eq!(manifest.resolve("geopotential", None), None); + } + + /// Out-of-range rejection: `facet = 3` is rejected. + #[test] + fn facet_out_of_range_is_rejected() { + let text = format!("{HEADER}\n3\t0\tlo\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::FacetOutOfRange { .. }) + )); + } + + /// Out-of-range rejection: `pair = 6` is rejected. + #[test] + fn pair_out_of_range_is_rejected() { + let text = format!("{HEADER}\n0\t6\tlo\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::PairOutOfRange { .. }) + )); + } + + /// Out-of-range rejection: a bogus byte token (`mid`) is rejected. + #[test] + fn bogus_byte_token_is_rejected() { + let text = format!("{HEADER}\n0\t0\tmid\tsome_var\tsurface\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::BadByteToken { .. }) + )); + } + + /// A malformed header is rejected rather than silently accepted with + /// misaligned columns. + #[test] + fn malformed_header_is_rejected() { + let text = "not\tthe\theader\n0\t0\tlo\tsome_var\tsurface\tPa\tid\n"; + assert!(matches!( + FieldManifest::parse(text), + Err(ManifestError::BadHeader { .. }) + )); + } + + /// A row with the wrong number of columns is rejected (`==`, not a + /// permissive minimum). + #[test] + fn wrong_column_count_is_rejected() { + let text = format!("{HEADER}\n0\t0\tlo\tsome_var\tsurface\tPa\n"); // 6 columns, not 7 + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::WrongColumnCount { line: 2, found: 6 }) + )); + } + + /// A `level_hpa` that is neither `surface` nor an integer is rejected. + #[test] + fn bad_level_token_is_rejected() { + let text = format!("{HEADER}\n0\t0\tlo\tsome_var\tnot_a_level\tPa\tid\n"); + assert!(matches!( + FieldManifest::parse(&text), + Err(ManifestError::BadLevel { .. }) + )); + } + + /// `#`-prefixed comment lines and blank lines are skipped wherever they + /// occur, not just before the header. + #[test] + fn comments_and_blank_lines_are_skipped_anywhere() { + let text = format!( + "# leading comment\n\n{HEADER}\n# a mid-file comment\n0\t0\tlo\tsome_var\tsurface\tPa\tid\n\n# trailing comment\n" + ); + let manifest = FieldManifest::parse(&text).expect("comments must not break parsing"); + assert_eq!(manifest.entries().len(), 1); + } +} diff --git a/probes/weather-p1/era5_variable_census.json b/probes/weather-p1/era5_variable_census.json new file mode 100644 index 000000000..13f41cc11 --- /dev/null +++ b/probes/weather-p1/era5_variable_census.json @@ -0,0 +1,75 @@ +{ + "coordinate_arrays_excluded": [ + "latitude", + "level", + "longitude", + "time" + ], + "counts": { + "levels": 13, + "pressure_level_fields": 91, + "pressure_level_variables": 7, + "static": 14, + "surface": 17, + "unclassified": 0 + }, + "derived": { + "bytes_per_timestep_512B_rows": 531578880, + "values_per_timestep": 126665280 + }, + "grid": { + "lat": 721, + "lon": 1440, + "n_cells": 1038240 + }, + "metadata_url": "https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr/.zmetadata", + "n_timesteps": 92044, + "pressure_level_variables": [ + "geopotential", + "specific_humidity", + "temperature", + "u_component_of_wind", + "v_component_of_wind", + "vertical_velocity", + "wind_speed" + ], + "probe": "era5_variable_census", + "static_variables": [ + "angle_of_sub_gridscale_orography", + "anisotropy_of_sub_gridscale_orography", + "geopotential_at_surface", + "high_vegetation_cover", + "lake_cover", + "lake_depth", + "land_sea_mask", + "low_vegetation_cover", + "slope_of_sub_gridscale_orography", + "soil_type", + "standard_deviation_of_filtered_subgrid_orography", + "standard_deviation_of_orography", + "type_of_high_vegetation", + "type_of_low_vegetation" + ], + "store_url": "https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr", + "surface_variables": [ + "10m_u_component_of_wind", + "10m_v_component_of_wind", + "10m_wind_speed", + "2m_temperature", + "mean_sea_level_pressure", + "sea_ice_cover", + "sea_surface_temperature", + "surface_pressure", + "toa_incident_solar_radiation", + "toa_incident_solar_radiation_12hr", + "toa_incident_solar_radiation_24hr", + "toa_incident_solar_radiation_6hr", + "total_cloud_cover", + "total_column_water_vapour", + "total_precipitation_12hr", + "total_precipitation_24hr", + "total_precipitation_6hr" + ], + "total_fields_per_cell": 122, + "unclassified_arrays": {} +} diff --git a/probes/weather-p1/era5_variable_census.py b/probes/weather-p1/era5_variable_census.py new file mode 100644 index 000000000..1dd9f0b62 --- /dev/null +++ b/probes/weather-p1/era5_variable_census.py @@ -0,0 +1,291 @@ +"""D-WXS-1a — the ERA5 WeatherBench2 Zarr variable census, as a committed, +re-runnable probe. + +`weather-soa-bake-v1.md` §0.6 asserts a variable census in prose (17 surface ++ 7 pressure-level x 13 levels = 91 + 14 static = 122 fields/cell, 92,044 +timesteps) with no artifact behind it. That is exactly the chat-only-figure +defect this arc has already caught three times (see the plan's §0 C1/⊘C3 +note: "a title match is not an existence check; list the bucket"). This +probe reads the store's own `.zmetadata` and emits the census as JSON, so +the numbers terminate at bytes fetched from the store rather than at a +sentence someone typed. + +Classification is by array shape rank, excluding the four coordinate +arrays (`time`, `level`, `latitude`, `longitude`): + + rank 3 [T, 721, 1440] -> time-varying SURFACE field + rank 4 [T, n_levels, 721, 1440] -> time-varying PRESSURE-LEVEL field + (contributes n_levels fields, one + per level) + rank 2 [721, 1440] -> STATIC field (no time axis) + +Anything that does not match one of those three shapes is reported under +`unclassified` rather than silently dropped or silently forced into a +bucket -- an unanticipated array shape is itself a finding. + +Run with `--selftest` to compare the fetched census against the constants +this plan already committed. A mismatch there means one of two things: +the store changed shape since the plan was written, or the plan's prose +was wrong. Either way the probe's own fetched answer wins -- that is the +point of terminating at the artifact instead of the prose. +""" + +import json +import pathlib +import sys +import urllib.error +import urllib.request + +STORE = ("https://storage.googleapis.com/weatherbench2/datasets/era5/" + "1959-2022-6h-1440x721.zarr") +METADATA_URL = STORE + "/.zmetadata" + +# The four coordinate arrays. Every other array in the store is a data +# variable (or an unanticipated shape, reported separately). +COORD_ARRAYS = {"time", "level", "latitude", "longitude"} + +HERE = pathlib.Path(__file__).parent +OUT = HERE / "era5_variable_census.json" + +# ---------------------------------------------------------------- fetch + +def _fetch_bytes(url, timeout=120): + """Fetch `url`, trying the default (proxied) opener first and falling + back to a proxy-bypassed opener on failure. + + This environment's outbound HTTPS goes through a pre-configured + agent proxy; a 403 or TLS error against a public GCS bucket is + usually the proxy, not the resource (see CLAUDE.md's own "GitHub + access matrix" finding, which generalizes past GitHub: verify with + the proxy bypassed before concluding a store is unreachable). + """ + try: + with urllib.request.urlopen(url, timeout=timeout) as resp: + return resp.read() + except Exception: # noqa: BLE001 -- broad on purpose, see docstring + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + with opener.open(url, timeout=timeout) as resp: + return resp.read() + + +def fetch_metadata(): + """Fetch and parse the store's consolidated `.zmetadata`. + + Returns the top-level `"metadata"` dict: keys are `"/.zarray"` + (plus `.zgroup` / `.zattrs` entries, which this probe ignores), values + carry `"shape"` and `"dtype"` among other zarr array-metadata fields. + """ + raw = _fetch_bytes(METADATA_URL) + doc = json.loads(raw) + return doc["metadata"] + + +# ------------------------------------------------------------ classify + +def array_entries(metadata): + """Yield (array_name, zarray_dict) for every `.zarray` entry in the + consolidated metadata, in the order the JSON gave them (not sorted -- + sorting happens once, at emission, on the classified name lists).""" + for key, val in metadata.items(): + if key.endswith("/.zarray"): + name = key[: -len("/.zarray")] + yield name, val + + +def classify(metadata): + """Classify every non-coordinate array by shape rank. + + Returns a dict with `surface`, `pressure_level` (name -> shape), + `static`, and `unclassified` (name -> shape, for anything that + doesn't match one of the three canonical shapes) plus the resolved + grid dims, timestep count, and level count pulled from the + coordinate arrays themselves (never hardcoded). + """ + coord_shapes = {} + for name, za in array_entries(metadata): + if name in COORD_ARRAYS: + coord_shapes[name] = za["shape"] + + missing = COORD_ARRAYS - set(coord_shapes) + if missing: + raise RuntimeError( + f"store is missing expected coordinate array(s): {sorted(missing)} " + "-- cannot classify without a time/level/lat/lon reference") + + n_time = coord_shapes["time"][0] + n_levels = coord_shapes["level"][0] + n_lat = coord_shapes["latitude"][0] + n_lon = coord_shapes["longitude"][0] + grid_shape = [n_lat, n_lon] + + surface, pressure_level, static, unclassified = {}, {}, {}, {} + + for name, za in array_entries(metadata): + if name in COORD_ARRAYS: + continue + shape = za["shape"] + rank = len(shape) + if rank == 2 and shape == grid_shape: + static[name] = shape + elif rank == 3 and shape[0] == n_time and shape[1:] == grid_shape: + surface[name] = shape + elif (rank == 4 and shape[0] == n_time and shape[1] == n_levels + and shape[2:] == grid_shape): + pressure_level[name] = shape + else: + unclassified[name] = shape + + return { + "surface": surface, + "pressure_level": pressure_level, + "static": static, + "unclassified": unclassified, + "n_time": n_time, + "n_levels": n_levels, + "n_lat": n_lat, + "n_lon": n_lon, + } + + +# --------------------------------------------------------------- report + +def build_report(metadata): + """Assemble the full committed-JSON census from a classified store.""" + c = classify(metadata) + + surface_names = sorted(c["surface"]) + pressure_names = sorted(c["pressure_level"]) + static_names = sorted(c["static"]) + unclassified_names = sorted(c["unclassified"]) + coord_names = sorted(COORD_ARRAYS) + + n_surface = len(surface_names) + n_pressure_vars = len(pressure_names) + n_levels = c["n_levels"] + n_pressure_fields = n_pressure_vars * n_levels + n_static = len(static_names) + total_fields_per_cell = n_surface + n_pressure_fields + n_static + + n_lat, n_lon = c["n_lat"], c["n_lon"] + n_cells = n_lat * n_lon + n_time = c["n_time"] + + values_per_timestep = n_cells * total_fields_per_cell + bytes_per_timestep_512b_rows = n_cells * 512 + + return { + "probe": "era5_variable_census", + "store_url": STORE, + "metadata_url": METADATA_URL, + "coordinate_arrays_excluded": coord_names, + "surface_variables": surface_names, + "pressure_level_variables": pressure_names, + "static_variables": static_names, + "unclassified_arrays": {n: c["unclassified"][n] for n in unclassified_names}, + "counts": { + "surface": n_surface, + "pressure_level_variables": n_pressure_vars, + "levels": n_levels, + "pressure_level_fields": n_pressure_fields, + "static": n_static, + "unclassified": len(unclassified_names), + }, + "total_fields_per_cell": total_fields_per_cell, + "grid": {"lat": n_lat, "lon": n_lon, "n_cells": n_cells}, + "n_timesteps": n_time, + "derived": { + "values_per_timestep": values_per_timestep, + "bytes_per_timestep_512B_rows": bytes_per_timestep_512b_rows, + }, + } + + +# --------------------------------------------------------------- selftest + +# The constants `weather-soa-bake-v1.md` §0.6 already committed in prose. +# A mismatch means either the store changed since the plan was written, or +# the plan's prose figure was wrong -- either way this probe's fetched +# answer is the one that wins, because it terminates at the artifact. +EXPECTED = { + "surface": 17, + "pressure_level_variables": 7, + "levels": 13, + "pressure_level_fields": 91, + "static": 14, + "total_fields_per_cell": 122, + "n_cells": 1_038_240, + "n_timesteps": 92_044, + "values_per_timestep": 126_665_280, + "bytes_per_timestep_512B_rows": 531_578_880, +} + + +def selftest(report): + """Compare the fetched report against EXPECTED. Fails loudly (exit 1) + and prints every mismatch on disagreement -- never adjusts EXPECTED + to make a run pass.""" + got = { + "surface": report["counts"]["surface"], + "pressure_level_variables": report["counts"]["pressure_level_variables"], + "levels": report["counts"]["levels"], + "pressure_level_fields": report["counts"]["pressure_level_fields"], + "static": report["counts"]["static"], + "total_fields_per_cell": report["total_fields_per_cell"], + "n_cells": report["grid"]["n_cells"], + "n_timesteps": report["n_timesteps"], + "values_per_timestep": report["derived"]["values_per_timestep"], + "bytes_per_timestep_512B_rows": + report["derived"]["bytes_per_timestep_512B_rows"], + } + mismatches = {k: (EXPECTED[k], got[k]) for k in EXPECTED if EXPECTED[k] != got[k]} + return mismatches, got + + +# -------------------------------------------------------------------- main + +def main(): + run_selftest = "--selftest" in sys.argv + + metadata = fetch_metadata() + report = build_report(metadata) + + OUT.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n") + + print(f"store: {STORE}") + print(f"metadata: {METADATA_URL}") + print(f"surface variables: {report['counts']['surface']}") + print(f"pressure-level variables: {report['counts']['pressure_level_variables']}" + f" x {report['counts']['levels']} levels" + f" = {report['counts']['pressure_level_fields']} fields") + print(f"static variables: {report['counts']['static']}") + print(f"unclassified arrays: {report['counts']['unclassified']}" + + (f" {sorted(report['unclassified_arrays'])}" + if report['unclassified_arrays'] else "")) + print(f"total fields/cell: {report['total_fields_per_cell']}") + print(f"grid: {report['grid']['lat']} x {report['grid']['lon']}" + f" = {report['grid']['n_cells']} cells") + print(f"timesteps: {report['n_timesteps']}") + print(f"values/timestep: {report['derived']['values_per_timestep']}") + print(f"bytes/timestep @512B rows: " + f"{report['derived']['bytes_per_timestep_512B_rows']}") + print(f"\nwrote {OUT}") + + if run_selftest: + mismatches, got = selftest(report) + if mismatches: + print("\nSELFTEST: FAIL -- fetched census disagrees with the plan's " + "committed constants (weather-soa-bake-v1.md §0.6):") + for k, (exp, act) in sorted(mismatches.items()): + print(f" {k:32s} expected={exp!r:>14} got={act!r:>14}") + print("\nThis means either the store changed shape since the plan " + "was written, or the plan's prose figure was wrong. The " + "fetched answer above is authoritative; do not edit EXPECTED " + "to silence this without updating the plan too.") + sys.exit(1) + else: + print(f"\nSELFTEST: PASS -- all {len(EXPECTED)} committed constants " + "match the fetched census.") + + +if __name__ == "__main__": + main() diff --git a/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt b/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt new file mode 100644 index 000000000..6198bc369 --- /dev/null +++ b/probes/weather-p1/exec-runs/substrate_comfort_d_cz_2_7.txt @@ -0,0 +1 @@ +DONE d-cz-2..7 diff --git a/probes/weather-p1/substrate_comfort_d_cz_2_7.json b/probes/weather-p1/substrate_comfort_d_cz_2_7.json new file mode 100644 index 000000000..bd3365a76 --- /dev/null +++ b/probes/weather-p1/substrate_comfort_d_cz_2_7.json @@ -0,0 +1,628 @@ +{ + "probe": "substrate_comfort_d_cz_2_7", + "seed": 20260812, + "timesteps": [ + 54358, + 55086, + 55818 + ], + "budget_cells_per_regime": 4225, + "units": { + "grad_p": "Pa/cell (flat, no cos-lat \u2014 the D-CZ-0 identified definition)", + "rmse": "Pa", + "bias": "Pa", + "decay_length": "grid cells", + "L": "dimensionless (rho difference)" + }, + "C0_controls": { + "R1_CALM": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R2_OCEAN": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R3_ACTIVE": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + }, + "R4_STORM": { + "CAL-SHUFFLE": true, + "GEO-DEGENERATE": true + } + }, + "C1_ladder": { + "per_timestep": { + "54358": { + "values": [ + 10.267982482910156, + 15.295616149902344, + 43.52193832397461, + 88.98800578870271 + ], + "ordered": true + }, + "55086": { + "values": [ + 8.420635223388672, + 20.382078170776367, + 21.0435848236084, + 88.98800578870271 + ], + "ordered": true + }, + "55818": { + "values": [ + 9.359210014343262, + 13.816144943237305, + 58.70938491821289, + 88.98800578870271 + ], + "ordered": true + } + }, + "holds_all": true, + "raw": { + "R1_CALM": { + "54358": 10.267982482910156, + "55086": 8.420635223388672, + "55818": 9.359210014343262 + }, + "R2_OCEAN": { + "54358": 15.295616149902344, + "55086": 20.382078170776367, + "55818": 13.816144943237305 + }, + "R3_ACTIVE": { + "54358": 43.52193832397461, + "55086": 21.0435848236084, + "55818": 58.70938491821289 + }, + "R4_STORM": { + "per_storm_mean": 88.98800578870271, + "per_storm_min": 39.901527404785156, + "n_independent_timesteps": 19 + } + } + }, + "C1b_separation": { + "per_timestep": { + "54358": 6.430169158055194, + "55086": 8.240830494144255, + "55818": 5.866366164820492 + }, + "bar": 3.0, + "holds_all": true + }, + "C1c_structure": { + "per_regime": { + "R1_CALM": { + "decay_length_cells": 25.5, + "gini_gradp": 0.4320934781503374, + "tail_ratio_gradp": 7.652638836611052 + }, + "R2_OCEAN": { + "decay_length_cells": 32.0, + "gini_gradp": 0.27581184507019185, + "tail_ratio_gradp": 3.722250017585821 + }, + "R3_ACTIVE": { + "decay_length_cells": 32.0, + "gini_gradp": 0.2262258291411534, + "tail_ratio_gradp": 1.9386696980814633 + }, + "R4_STORM": { + "decay_length_cells": 22.5, + "gini_gradp": 0.3062475714591397, + "tail_ratio_gradp": 2.9491883985991496, + "n": 19 + } + }, + "verdict": { + "decay_ratio_r4_over_r1": 0.8823529411764706, + "gini_ratio_r4_over_r1": 0.7087530521638368, + "tail_ratio_r4_over_r1": 0.3853818874203122, + "distinguishable": true + } + }, + "C2_degenerate_row": { + "dynamic_rows_flat": { + "CAL-RANK": { + "R1_CALM": 0.0, + "R2_OCEAN": 0.0, + "R3_ACTIVE": 0.0, + "R4_STORM": 0.0 + }, + "CAL-FISHERZ": { + "R1_CALM": 0.0, + "R2_OCEAN": 0.0, + "R3_ACTIVE": 0.0, + "R4_STORM": 0.0 + } + }, + "cal_abs_can_differ": { + "R1_CALM": 0.01103689701168109, + "R2_OCEAN": 0.6183197101240259, + "R3_ACTIVE": 0.8343471141874098, + "R4_STORM": 0.9467064832844216 + }, + "cal_abs_undefined_donors": { + "R1_CALM": [ + "R2_OCEAN", + "R3_ACTIVE" + ], + "R2_OCEAN": [ + "R1_CALM" + ], + "R3_ACTIVE": [ + "R1_CALM" + ], + "R4_STORM": [] + }, + "half_i_dynamic_L_is_exactly_zero": true, + "half_ii_cal_abs_nonzero_somewhere": true + }, + "C3_transfer_loss": { + "per_regime": { + "R1_CALM": { + "L_bar": 0.01103689701168109, + "n_defined_offdiag": 1, + "n_undefined_offdiag": 2, + "occupancy_mean_offdiag": 0.022135416666666668, + "saturation_mean_offdiag": 0.6666666666666666 + }, + "R2_OCEAN": { + "L_bar": 0.3094828671426827, + "n_defined_offdiag": 2, + "n_undefined_offdiag": 1, + "occupancy_mean_offdiag": 0.08463541666666667, + "saturation_mean_offdiag": 0.6609072978303747 + }, + "R3_ACTIVE": { + "L_bar": 0.6708209545588866, + "n_defined_offdiag": 2, + "n_undefined_offdiag": 1, + "occupancy_mean_offdiag": 0.10026041666666667, + "saturation_mean_offdiag": 0.9674161735700197 + }, + "R4_STORM": { + "L_bar": 0.6899068233087143, + "n_defined_offdiag": 3, + "n_undefined_offdiag": 0, + "occupancy_mean_offdiag": 0.3411458333333333, + "saturation_mean_offdiag": 0.9686785009861932 + } + }, + "r1_r4_both_defined": true, + "holds_r4_lt_r1": false + }, + "C4_crossover": { + "per_regime": { + "R1_CALM": { + "delta_rmse_rank_minus_absdiag": 1.2875496387398329, + "rho_floor_check_min": 0.99991523929089, + "weak_form_absforeign_minus_rank_rmse": 439.45736416501524 + }, + "R2_OCEAN": { + "delta_rmse_rank_minus_absdiag": 1.110984110781117, + "rho_floor_check_min": 0.999916685049489, + "weak_form_absforeign_minus_rank_rmse": 328.7326356721833 + }, + "R3_ACTIVE": { + "delta_rmse_rank_minus_absdiag": 0.4552001128419074, + "rho_floor_check_min": 0.9999168206445191, + "weak_form_absforeign_minus_rank_rmse": 1635.2923826909162 + }, + "R4_STORM": { + "delta_rmse_rank_minus_absdiag": 10.780337293854465, + "rho_floor_check_min": 0.9999168302772791, + "weak_form_absforeign_minus_rank_rmse": 2507.545438988307 + } + }, + "sign_flip": false + }, + "C6_matrix": { + "R1_CALM": { + "CAL-ABS|R1_CALM": { + "rho": 0.9999596603993293, + "rmse_pa": 0.43545659099220974, + "bias_pa": -0.0039270091762204145, + "occupancy": 0.8359375, + "saturation": 0.00047337278106508875 + }, + "CAL-ABS|R2_OCEAN": { + "rho": NaN, + "rmse_pa": 245.86851734876652, + "bias_pa": 237.65027211567354, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": NaN, + "rmse_pa": 1069.5368025036043, + "bias_pa": 1067.677524312939, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9889227633876482, + "rmse_pa": 8.135791331870866, + "bias_pa": 0.19725649500739645, + "occupancy": 0.05859375, + "saturation": 0.0 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999927605048448, + "rmse_pa": 1.7230062297320425, + "bias_pa": 0.19835151627218936, + "occupancy": 1.0, + "saturation": 0.0078106508875739646 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.99991523929089, + "rmse_pa": 0.8066562813008817, + "bias_pa": 0.02618897928994083, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.08232528142572652, + "rmse_pa": 140.96730416561363, + "bias_pa": 59.71426860380455, + "occupancy": 0.8359375, + "saturation": 0.00047337278106508875 + }, + "GEO-DEGENERATE|self": { + "rho": 0.3799822353435944, + "rmse_pa": 119.99599105198327, + "bias_pa": -103.18710295727972, + "occupancy": 0.421875, + "saturation": 0.949585798816568 + } + }, + "R2_OCEAN": { + "CAL-ABS|R1_CALM": { + "rho": NaN, + "rmse_pa": 525.9022909934547, + "bias_pa": -463.10123923758783, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.9999805198027167, + "rmse_pa": 1.3133243896042126, + "bias_pa": 0.02958709175064719, + "occupancy": 0.9609375, + "saturation": 0.0009467455621301775 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.3816608096786909, + "rmse_pa": 445.52070839237126, + "bias_pa": 376.5969895438471, + "occupancy": 0.1171875, + "saturation": 0.9488757396449704 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9993344956413772, + "rmse_pa": 22.047833131879898, + "bias_pa": -2.914080875554734, + "occupancy": 0.1328125, + "saturation": 0.033846153846153845 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999923760691042, + "rmse_pa": 2.4243085003853295, + "bias_pa": 0.1679363905325444, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.999916685049489, + "rmse_pa": 3.2512974601170033, + "bias_pa": -0.06652551775147929, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": -0.04867588197240352, + "rmse_pa": 452.1981665483234, + "bias_pa": 122.80125530897513, + "occupancy": 0.9609375, + "saturation": 0.0009467455621301775 + }, + "GEO-DEGENERATE|self": { + "rho": 0.47801853597161725, + "rmse_pa": 430.5506483061611, + "bias_pa": -354.5808563629692, + "occupancy": 0.6484375, + "saturation": 0.917396449704142 + } + }, + "R3_ACTIVE": { + "CAL-ABS|R1_CALM": { + "rho": NaN, + "rmse_pa": 2258.161742708387, + "bias_pa": -2129.5621933795996, + "occupancy": 0.00390625, + "saturation": 1.0 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.4926944507153999, + "rmse_pa": 1224.020331977599, + "bias_pa": -979.967107865373, + "occupancy": 0.28125, + "saturation": 0.9114792899408284 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.9999892456457632, + "rmse_pa": 3.2888446705759047, + "bias_pa": -0.1164641358866494, + "occupancy": 1.0, + "saturation": 0.002366863905325444 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.1656421314583534, + "rmse_pa": 1434.9272077370156, + "bias_pa": -1223.0108273622411, + "occupancy": 0.015625, + "saturation": 0.9907692307692307 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.99999238780557, + "rmse_pa": 3.744044783417812, + "bias_pa": 0.30272697855029584, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.9999168206445191, + "rmse_pa": 9.686523384544712, + "bias_pa": 0.09407729289940829, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.1076028284432391, + "rmse_pa": 1056.1259072749588, + "bias_pa": 151.59143322103827, + "occupancy": 1.0, + "saturation": 0.002366863905325444 + }, + "GEO-DEGENERATE|self": { + "rho": 0.2900859787206164, + "rmse_pa": 1594.7041920434797, + "bias_pa": 1408.846735254773, + "occupancy": 0.3359375, + "saturation": 0.9718343195266272 + } + }, + "R4_STORM": { + "CAL-ABS|R1_CALM": { + "rho": 0.5072287551023801, + "rmse_pa": 2070.4683206201353, + "bias_pa": 1647.7540435321514, + "occupancy": 0.56640625, + "saturation": 0.954792899408284 + }, + "CAL-ABS|R2_OCEAN": { + "rho": 0.36970751728458107, + "rmse_pa": 2388.553976512638, + "bias_pa": 2021.596971074392, + "occupancy": 0.4375, + "saturation": 0.952189349112426 + }, + "CAL-ABS|R3_ACTIVE": { + "rho": 0.053268646229919614, + "rmse_pa": 3119.8878118239486, + "bias_pa": 2838.3526022068, + "occupancy": 0.01953125, + "saturation": 0.9990532544378699 + }, + "CAL-ABS|R4_STORM": { + "rho": 0.9999751295143413, + "rmse_pa": 7.977593370078924, + "bias_pa": 0.10601527829142011, + "occupancy": 0.94140625, + "saturation": 0.0009467455621301775 + }, + "CAL-RANK|R1_CALM": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R1_CALM": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R2_OCEAN": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R2_OCEAN": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R3_ACTIVE": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R3_ACTIVE": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-RANK|R4_STORM": { + "rho": 0.9999923744008613, + "rmse_pa": 18.75793066393339, + "bias_pa": 0.0618958025147929, + "occupancy": 1.0, + "saturation": 0.008047337278106508 + }, + "CAL-FISHERZ|R4_STORM": { + "rho": 0.9999168302772791, + "rmse_pa": 16.183609077920757, + "bias_pa": 0.023818417159763315, + "occupancy": 0.7421875, + "saturation": 0.00047337278106508875 + }, + "CAL-SHUFFLE|self": { + "rho": 0.006699959537369512, + "rmse_pa": 2536.695394356023, + "bias_pa": -603.1110531850961, + "occupancy": 0.94140625, + "saturation": 0.0009467455621301775 + }, + "GEO-DEGENERATE|self": { + "rho": 0.9038053970526956, + "rmse_pa": 1196.4020646066267, + "bias_pa": 420.62941414917714, + "occupancy": 0.921875, + "saturation": 0.8435502958579881 + } + } + }, + "c5_structural_blocker": "C5 (GEO-GOLDEN-LO vs GEO-GOLDEN-HI) is NOT RUN, and the reason is structural rather than budgetary. The golden index floor requires N >= F(17)^2 = 2,550,409 lattice points before the emergent parastichy stride behaves like phi at all. A 16-degree box at 0.25 degree resolution contains 65 x 65 = 4,225 cells -- three orders of magnitude below the floor. GEO-GOLDEN-HI therefore CANNOT be constructed at box scale on this grid, so the comparison the bar asks for has no admissible high arm. Reported rather than faked with an interpolated sub-grid lattice, which would sample the interpolator and not the field." +} \ No newline at end of file diff --git a/probes/weather-p1/substrate_comfort_d_cz_2_7.py b/probes/weather-p1/substrate_comfort_d_cz_2_7.py new file mode 100644 index 000000000..326791d68 --- /dev/null +++ b/probes/weather-p1/substrate_comfort_d_cz_2_7.py @@ -0,0 +1,499 @@ +"""D-CZ-2..7 — the cross-swap transfer matrix and the gates that precede it. + +Runs the bars pre-registered in `.claude/plans/substrate-comfort-zones-v1.md` +§3, in the order the plan itself requires: **C1c first**, because a null there +VOIDS the cross-swap interpretation rather than weakening it. + + C1 regime ladder holds on >= 3 independent timesteps + C1b separation = (between-box range) / (mean within-box sigma) >= 3 + C1c the regimes must differ in CORRELATION STRUCTURE, not merely |grad p| + C2 the degenerate row -- both halves (dynamic L == 0 exactly, AND + CAL-ABS proven able to be non-degenerate through the same code path) + C3 transfer loss shrinks with turbulence + C4 the crossover, on the AMENDED metric (RMSE in Pa, not rho -- §6.4) + C6 the full transfer matrix, every cell raw + +C5 (the geometry axis) is NOT run here, and the reason is structural rather +than budgetary -- see `c5_structural_blocker` in the JSON. It is reported, +not skipped silently. + +EQUAL BUDGET is enforced by construction: every regime is subsampled to the +same cell count with a seeded RNG before any codebook is built. The W2s-a +lesson (an unequal budget silently advantages the arm with more samples) is +not re-learned here. + +R4's timesteps are the 19 storms' OWN t0 values -- measuring a storm box at +an unrelated hour is the defect this probe's predecessor already caught and +fixed. That also means R4 satisfies C1's ">= 3 independent timesteps" with +19, while R1..R3 are measured at 3 explicit timesteps each. +""" + +import datetime +import json +import pathlib +import urllib.request + +import numcodecs +import numpy as np + +SEED = 20260812 +B = ("https://storage.googleapis.com/weatherbench2/datasets/era5/" + "1959-2022-6h-1440x721.zarr") +R_E = 6371.0 +BOX_DEG = 16.0 +N_LEVELS = 256 +G0 = 9.80665 + +# C1 needs >= 3 independent timesteps. t0 is the preflight's; the other two are +# offset by ~6 months and ~1 year so the ladder is tested across seasons, not +# across three adjacent hours (which would be one condition sampled thrice). +TIMESTEPS = [54358, 54358 + 4 * 182, 54358 + 4 * 365] + +HERE = pathlib.Path(__file__).parent +OUT = HERE / "substrate_comfort_d_cz_2_7.json" +TAG = HERE / "exec-runs" / "substrate_comfort_d_cz_2_7.txt" + +op = urllib.request.build_opener(urllib.request.ProxyHandler({})) +meta = json.loads(op.open(B + "/.zmetadata", timeout=90).read())["metadata"] +EPOCH = datetime.datetime(1959, 1, 1) + + +def t_index(dt): + """WB2 time index: 6-hourly steps since 1959-01-01. Anchor guard.""" + return int(round((dt - EPOCH).total_seconds() / 3600 / 6)) + + +assert t_index(datetime.datetime(2021, 6, 15, 12)) == 91246 + + +def fetch(var, key): + """Fetch and decode one zarr chunk from the WB2 store.""" + za = meta[f"{var}/.zarray"] + raw = op.open(f"{B}/{var}/{key}", timeout=900).read() + dec = numcodecs.get_codec(za["compressor"]).decode(raw) + return np.frombuffer(dec, dtype=np.dtype(za["dtype"])).reshape(za["chunks"]) + + +LATS = np.linspace(90.0, -90.0, 721) +LONS = np.linspace(0.0, 359.75, 1440) + + +def box_index(clat, clon, deg=BOX_DEG): + """Row indices and WRAPPED column indices for a deg x deg box.""" + i0 = int(np.argmin(np.abs(LATS - (clat + deg / 2)))) + i1 = int(np.argmin(np.abs(LATS - (clat - deg / 2)))) + jc = int(np.argmin(np.abs(LONS - (clon % 360)))) + half = int(round((deg / 2) / 0.25)) + return np.arange(i0, i1 + 1), (np.arange(jc - half, jc + half + 1)) % len(LONS) + + +def box(field, rows, cols): + """Extract a box from a 2-D field, honouring the longitude wrap.""" + return field[np.ix_(rows, cols)] + + +def grad_flat(p, rows, cols): + """|grad p| in Pa per grid cell, NO cos(lat) — the definition identified + from the data in D-CZ-0 as the one the §1 preflight figures used. Kept so + the ladder here is comparable to the recorded table, not to a different + quantity.""" + sub = box(p, rows, cols) + return np.hypot(np.gradient(sub, axis=1), np.gradient(sub, axis=0)) + + +def spearman(a, b): + """Spearman rho via Pearson on ranks, ties averaged; nan on constant + input (rho is undefined there, not zero).""" + def rank(x): + order = np.argsort(x, kind="stable") + r = np.empty(len(x), float) + r[order] = np.arange(len(x), dtype=float) + xs = x[order] + i = 0 + while i < len(xs): + j = i + while j + 1 < len(xs) and xs[j + 1] == xs[i]: + j += 1 + if j > i: + r[order[i:j + 1]] = (i + j) / 2.0 + i = j + 1 + return r + ra, rb = rank(np.asarray(a, float)), rank(np.asarray(b, float)) + if ra.std() == 0 or rb.std() == 0: + return float("nan") + return float(np.corrcoef(ra, rb)[0, 1]) + + +# ------------------------------------------------------- C1c instruments + +def decay_length_cells(field2d): + """Autocorrelation decay length in grid cells: the lag at which the + field's spatial autocorrelation first drops below 1/e, averaged over the + two axes. Measured on the ANOMALY (box mean removed) so the decay is of + structure, not of the offset.""" + a = field2d - field2d.mean() + out = [] + for axis in (0, 1): + n = a.shape[axis] + cors = [] + for lag in range(1, min(n // 2, 40)): + x = np.take(a, range(0, n - lag), axis=axis) + y = np.take(a, range(lag, n), axis=axis) + xv, yv = x.ravel(), y.ravel() + if xv.std() == 0 or yv.std() == 0: + cors.append(0.0) + else: + cors.append(float(np.corrcoef(xv, yv)[0, 1])) + thr = 1.0 / np.e + hit = next((i + 1 for i, c in enumerate(cors) if c < thr), len(cors) + 1) + out.append(float(hit)) + return float(np.mean(out)) + + +def gini(x): + """Gini coefficient of a non-negative sample — the concentration of the + |grad p| distribution. 0 = every cell equally gradient-y, 1 = all the + gradient in one cell.""" + v = np.sort(np.asarray(x, float).ravel()) + v = v[np.isfinite(v)] + if v.size == 0 or v.sum() == 0: + return float("nan") + n = v.size + idx = np.arange(1, n + 1) + return float((2 * (idx * v).sum()) / (n * v.sum()) - (n + 1) / n) + + +def tail_ratio(x): + """p99 / p50 of |grad p| — how far the extreme sits above the typical.""" + v = np.asarray(x, float).ravel() + med = float(np.percentile(v, 50)) + return float(np.percentile(v, 99) / med) if med > 0 else float("nan") + + +# ------------------------------------------------------------- the codecs + +def encode_decode(values, lo, hi, decode_table=None): + """Quantize to N_LEVELS uniform levels over [lo, hi], decode to the level + centre. Values outside SATURATE — the mechanism a wrong donor fails by.""" + lo, hi = float(lo), float(hi) + if hi <= lo: + hi = lo + 1e-9 + width = (hi - lo) / N_LEVELS + idx = np.clip(((values - lo) / width).astype(int), 0, N_LEVELS - 1) + centres = lo + (np.arange(N_LEVELS) + 0.5) * width + if decode_table is not None: + centres = centres[decode_table] + return centres[idx], idx + + +def rank_codec(values): + """CAL-RANK: 256 quantile levels re-derived INSIDE the window. No donor + exists — that is the property under test, not a defect.""" + qs = np.quantile(values, np.linspace(0, 1, N_LEVELS + 1)) + idx = np.clip(np.searchsorted(qs, values, side="right") - 1, 0, N_LEVELS - 1) + return 0.5 * (qs[:-1] + qs[1:])[idx], idx + + +def fisherz_codec(values): + """CAL-FISHERZ: uniform levels in arctanh(2*rank-1) — resolution moved + into the tails. Also window-local, so also donor-free.""" + n = len(values) + r = (np.argsort(np.argsort(values)) + 0.5) / n + s = np.clip(2 * r - 1, -0.999999, 0.999999) + z = np.arctanh(s) + dec_z, idx = encode_decode(z, z.min(), z.max()) + # map the decoded z back to a value by inverting through the sorted sample + back = np.clip((np.tanh(dec_z) + 1) / 2, 0, 1) + srt = np.sort(values) + pos = np.clip((back * n).astype(int), 0, n - 1) + return srt[pos], idx + + +def cell_metrics(dec, idx, truth): + """One matrix cell's numbers. Every quantity the plan's C6 requires.""" + resid = dec - truth + return { + "rho": spearman(dec, truth), + "rmse_pa": float(np.sqrt(np.mean(resid ** 2))), + "bias_pa": float(np.mean(resid)), + "occupancy": float(len(np.unique(idx)) / N_LEVELS), + "saturation": float(np.mean((idx == 0) | (idx == N_LEVELS - 1))), + } + + +# ------------------------------------------------------------------- main + +def main(): + """Run C1c first (it can void everything), then C1/C1b, then the matrix.""" + TAG.parent.mkdir(exist_ok=True) + TAG.write_text("START d-cz-2..7\n") + rng = np.random.default_rng(SEED) + + named = { + "R1_CALM": (-4.0, 296.0), + "R2_OCEAN": (-25.0, 220.0), + "R3_ACTIVE": (60.0, 72.0), + } + f14 = json.loads((HERE / "comet_tail_f14.json").read_text()) + qual = [r for r in f14["rows"] + if r.get("status") == "OK" and r.get("displacement_km", 0) >= 250] + assert len(qual) == f14["n_qualifying"] == 19 + + # ---- fields ---- + fields = {t: fetch("mean_sea_level_pressure", f"{t}.0.0")[0] for t in TIMESTEPS} + storm_fields = {} + for r in qual: + t = int(r["t0"]) + if t not in storm_fields: + storm_fields[t] = fetch("mean_sea_level_pressure", f"{t}.0.0")[0] + + # ---- C1 / C1b : ladder + separation across timesteps ---- + ladder, within = {}, {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + ladder[name] = {str(t): float(grad_flat(fields[t], rr, cc).mean()) + for t in TIMESTEPS} + within[name] = {str(t): float(grad_flat(fields[t], rr, cc).std()) + for t in TIMESTEPS} + storm_g = [] + for r in qual: + rr, cc = box_index(float(r["center_lat"]), float(r["center_lon"])) + storm_g.append(float(grad_flat(storm_fields[int(r["t0"])], rr, cc).mean())) + ladder["R4_STORM"] = {"per_storm_mean": float(np.mean(storm_g)), + "per_storm_min": float(np.min(storm_g)), + "n_independent_timesteps": len(set(int(r["t0"]) for r in qual))} + + c1 = {} + for t in TIMESTEPS: + vals = [ladder["R1_CALM"][str(t)], ladder["R2_OCEAN"][str(t)], + ladder["R3_ACTIVE"][str(t)], float(np.mean(storm_g))] + c1[str(t)] = {"values": vals, "ordered": bool(all( + vals[i] < vals[i + 1] for i in range(3)))} + c1_holds = all(v["ordered"] for v in c1.values()) + + c1b = {} + for t in TIMESTEPS: + means = [ladder[k][str(t)] for k in named] + [float(np.mean(storm_g))] + sig = float(np.mean([within[k][str(t)] for k in named])) + c1b[str(t)] = float((max(means) - min(means)) / sig) + + # ---- C1c : do the regimes differ in STRUCTURE, not just in |grad p|? ---- + c1c = {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + f = box(fields[TIMESTEPS[0]], rr, cc) + g = grad_flat(fields[TIMESTEPS[0]], rr, cc) + c1c[name] = {"decay_length_cells": decay_length_cells(f), + "gini_gradp": gini(g), "tail_ratio_gradp": tail_ratio(g)} + dl, gi, tr = [], [], [] + for r in qual: + rr, cc = box_index(float(r["center_lat"]), float(r["center_lon"])) + f = box(storm_fields[int(r["t0"])], rr, cc) + g = grad_flat(storm_fields[int(r["t0"])], rr, cc) + dl.append(decay_length_cells(f)); gi.append(gini(g)); tr.append(tail_ratio(g)) + c1c["R4_STORM"] = {"decay_length_cells": float(np.median(dl)), + "gini_gradp": float(np.median(gi)), + "tail_ratio_gradp": float(np.median(tr)), + "n": len(dl)} + # the bar: R1 and R4 must be DISTINGUISHABLE on these + r1, r4 = c1c["R1_CALM"], c1c["R4_STORM"] + c1c_verdict = { + "decay_ratio_r4_over_r1": r4["decay_length_cells"] / r1["decay_length_cells"], + "gini_ratio_r4_over_r1": r4["gini_gradp"] / r1["gini_gradp"], + "tail_ratio_r4_over_r1": r4["tail_ratio_gradp"] / r1["tail_ratio_gradp"], + } + c1c_verdict["distinguishable"] = bool( + max(abs(v - 1.0) for k, v in c1c_verdict.items() if k.endswith("r1")) >= 0.20) + + # ---- equal-budget samples, one per regime -- AND a genuine spatial + # sub-patch per regime for GEO-DEGENERATE, kept SEPARATE from the + # equal-budget flat sample. Slicing the flat (post-subsample, post- + # shuffle) array is NOT a spatial sub-patch -- rng.choice returns no + # spatial order, so its first N elements are an ordinary random + # subsample, not a narrow corner. The degenerate donor must be built + # from the box's own 2-D layout, exactly as D-CZ-1 does it. + boxes2d = {} + for name, (clat, clon) in named.items(): + rr, cc = box_index(clat, clon) + boxes2d[name] = box(fields[TIMESTEPS[0]], rr, cc).astype(float) + storm_boxes2d = [ + box(storm_fields[int(r["t0"])], + *box_index(float(r["center_lat"]), float(r["center_lon"]))).astype(float) + for r in qual] + # R4's "box" for the degenerate patch is the storm whose |grad p| is the + # tier median, matching D-CZ-1's own choice for the same purpose. + boxes2d["R4_STORM"] = storm_boxes2d[int(np.argsort(storm_g)[len(storm_g) // 2])] + + samples = {k: v.ravel() for k, v in boxes2d.items()} + samples["R4_STORM"] = np.concatenate( + [b.ravel() for b in storm_boxes2d]).astype(float) + budget = min(len(v) for v in samples.values()) + for k in list(samples): + if len(samples[k]) > budget: + samples[k] = samples[k][rng.choice(len(samples[k]), budget, replace=False)] + assert len({len(v) for v in samples.values()}) == 1, "equal budget violated" + + def degenerate_patch_range(name): + """A real spatial corner -- n_i x n_i cells, n_i = max(1, side//8) -- + matching D-CZ-1's `p[si,sj][:n_i,:n_i]` construction exactly.""" + b2 = boxes2d[name] + n_i = max(1, b2.shape[0] // 8) + patch = b2[:n_i, :n_i] + return float(patch.min()), float(patch.max()) + + REG = list(samples) + + # ---- C2 / C3 / C4 / C6 : the transfer matrix ---- + matrix = {} + for T in REG: + truth = samples[T] + matrix[T] = {} + for D in REG: # CAL-ABS: a REAL donor matrix + dec, idx = encode_decode(truth, samples[D].min(), samples[D].max()) + matrix[T][f"CAL-ABS|{D}"] = cell_metrics(dec, idx, truth) + for D in REG: # dynamic arms: donor is IGNORED by + dec, idx = rank_codec(truth) # construction — that is the test + matrix[T][f"CAL-RANK|{D}"] = cell_metrics(dec, idx, truth) + dec, idx = fisherz_codec(truth) + matrix[T][f"CAL-FISHERZ|{D}"] = cell_metrics(dec, idx, truth) + perm = rng.permutation(N_LEVELS) # controls, on the diagonal + dec, idx = encode_decode(truth, truth.min(), truth.max(), decode_table=perm) + matrix[T]["CAL-SHUFFLE|self"] = cell_metrics(dec, idx, truth) + plo, phi = degenerate_patch_range(T) + dec, idx = encode_decode(truth, plo, phi) + matrix[T]["GEO-DEGENERATE|self"] = cell_metrics(dec, idx, truth) + + def L(arm, D, T): + return matrix[T][f"{arm}|{T}"]["rho"] - matrix[T][f"{arm}|{D}"]["rho"] + + def undefined_donors(T): + """Donors under which CAL-ABS's decode of T is fully saturated to one + level -- rho is undefined there, not merely small. A first-class + outcome, not a missing measurement: it means the donor's range is + so narrow relative to the target that structure is destroyed + entirely, the extreme end of what L is built to detect.""" + return [D for D in REG if np.isnan(matrix[T][f"CAL-ABS|{D}"]["rho"])] + + c2 = {"dynamic_rows_flat": {}, "cal_abs_can_differ": {}, + "cal_abs_undefined_donors": {T: undefined_donors(T) for T in REG}} + for arm in ("CAL-RANK", "CAL-FISHERZ"): + # dynamic arms have no donor at all, so L is 0 vs every D by + # construction -- never nan; a plain max is correct here. + c2["dynamic_rows_flat"][arm] = { + T: max(abs(L(arm, D, T)) for D in REG) for T in REG} + for T in REG: + defined = [abs(L("CAL-ABS", D, T)) for D in REG if D not in undefined_donors(T)] + c2["cal_abs_can_differ"][T] = max(defined) if defined else float("nan") + c2["half_i_dynamic_L_is_exactly_zero"] = bool(all( + v == 0.0 for a in c2["dynamic_rows_flat"].values() for v in a.values())) + c2["half_ii_cal_abs_nonzero_somewhere"] = bool( + np.nanmax(list(c2["cal_abs_can_differ"].values())) > 0) + + def l_bar(T): + """Mean off-diagonal transfer loss over DEFINED donors, with the + undefined (fully-saturated) donors counted separately rather than + silently dropped or silently nan-poisoning the mean.""" + vals = [L("CAL-ABS", D, T) for D in REG if D != T + and not np.isnan(matrix[T][f"CAL-ABS|{D}"]["rho"])] + return (float(np.mean(vals)) if vals else float("nan")), len(vals) + + c3 = {} + for T in REG: + lb, n_defined = l_bar(T) + c3[T] = {"L_bar": lb, "n_defined_offdiag": n_defined, + "n_undefined_offdiag": len(REG) - 1 - n_defined, + "occupancy_mean_offdiag": float(np.mean( + [matrix[T][f"CAL-ABS|{D}"]["occupancy"] for D in REG if D != T])), + "saturation_mean_offdiag": float(np.mean( + [matrix[T][f"CAL-ABS|{D}"]["saturation"] for D in REG if D != T]))} + # C3 as pre-registered compares L_bar(R4) to L_bar(R1). Both must be + # DEFINED for the comparison to mean what the bar says; report which + # case obtains rather than silently computing a NaN-laundered verdict. + c3_r1_r4_comparable = bool(np.isfinite(c3["R1_CALM"]["L_bar"]) + and np.isfinite(c3["R4_STORM"]["L_bar"])) + c3_holds = (bool(c3["R4_STORM"]["L_bar"] < c3["R1_CALM"]["L_bar"]) + if c3_r1_r4_comparable else None) + + c4 = {} + for T in REG: + d_rmse = (matrix[T][f"CAL-RANK|{T}"]["rmse_pa"] + - matrix[T][f"CAL-ABS|{T}"]["rmse_pa"]) + weak = float(np.mean([matrix[T][f"CAL-ABS|{D}"]["rmse_pa"] + for D in REG if D != T])) - matrix[T][f"CAL-RANK|{T}"]["rmse_pa"] + c4[T] = {"delta_rmse_rank_minus_absdiag": d_rmse, + "rho_floor_check_min": min(matrix[T][f"CAL-{a}|{T}"]["rho"] + for a in ("ABS", "RANK", "FISHERZ")), + "weak_form_absforeign_minus_rank_rmse": weak} + c4_flip = bool(c4["R1_CALM"]["delta_rmse_rank_minus_absdiag"] > 0 + and c4["R2_OCEAN"]["delta_rmse_rank_minus_absdiag"] > 0 + and c4["R4_STORM"]["delta_rmse_rank_minus_absdiag"] < 0) + + c0 = {} + for T in REG: + real = min(matrix[T][f"CAL-{a}|{T}"]["rho"] for a in ("ABS", "RANK", "FISHERZ")) + c0[T] = {c: bool(matrix[T][f"{c}|self"]["rho"] < real) + for c in ("CAL-SHUFFLE", "GEO-DEGENERATE")} + + result = { + "probe": "substrate_comfort_d_cz_2_7", "seed": SEED, + "timesteps": TIMESTEPS, "budget_cells_per_regime": int(budget), + "units": {"grad_p": "Pa/cell (flat, no cos-lat — the D-CZ-0 identified " + "definition)", "rmse": "Pa", "bias": "Pa", + "decay_length": "grid cells", "L": "dimensionless (rho difference)"}, + "C0_controls": c0, + "C1_ladder": {"per_timestep": c1, "holds_all": bool(c1_holds), + "raw": ladder}, + "C1b_separation": {"per_timestep": c1b, "bar": 3.0, + "holds_all": bool(all(v >= 3.0 for v in c1b.values()))}, + "C1c_structure": {"per_regime": c1c, "verdict": c1c_verdict}, + "C2_degenerate_row": c2, + "C3_transfer_loss": {"per_regime": c3, + "r1_r4_both_defined": c3_r1_r4_comparable, + "holds_r4_lt_r1": c3_holds}, + "C4_crossover": {"per_regime": c4, "sign_flip": bool(c4_flip)}, + "C6_matrix": matrix, + "c5_structural_blocker": ( + "C5 (GEO-GOLDEN-LO vs GEO-GOLDEN-HI) is NOT RUN, and the reason is " + "structural rather than budgetary. The golden index floor requires " + "N >= F(17)^2 = 2,550,409 lattice points before the emergent " + "parastichy stride behaves like phi at all. A 16-degree box at 0.25 " + "degree resolution contains 65 x 65 = 4,225 cells -- three orders of " + "magnitude below the floor. GEO-GOLDEN-HI therefore CANNOT be " + "constructed at box scale on this grid, so the comparison the bar " + "asks for has no admissible high arm. Reported rather than faked " + "with an interpolated sub-grid lattice, which would sample the " + "interpolator and not the field."), + } + OUT.write_text(json.dumps(result, indent=2)) + TAG.write_text("DONE d-cz-2..7\n") + + print(f"budget/regime = {budget} cells\n") + print("C1c STRUCTURE (runs first — a null here VOIDS the reading):") + for k, v in c1c.items(): + print(f" {k:10s} decay={v['decay_length_cells']:5.2f} cells " + f"gini={v['gini_gradp']:.4f} tail={v['tail_ratio_gradp']:.3f}") + print(f" R4/R1 ratios: {c1c_verdict}") + print(f"\nC1 ladder holds all timesteps: {c1_holds}") + print(f"C1b separation per timestep: " + f"{ {k: round(v,2) for k,v in c1b.items()} }") + print(f"\nC2 half(i) dynamic L==0 exactly: {c2['half_i_dynamic_L_is_exactly_zero']}") + print(f"C2 half(ii) CAL-ABS non-degenerate: {c2['half_ii_cal_abs_nonzero_somewhere']}" + f" max|L| per regime: { {k: round(v,4) for k,v in c2['cal_abs_can_differ'].items()} }") + print("\nC3 transfer loss L_bar (lower = travels better; " + "undefined = donor saturated the target to one level):") + for k, v in c3.items(): + lb = f"{v['L_bar']:+.5f}" if np.isfinite(v['L_bar']) else "UNDEFINED" + print(f" {k:10s} L_bar={lb:>10s} defined={v['n_defined_offdiag']}/" + f"{v['n_defined_offdiag']+v['n_undefined_offdiag']} " + f"occ={v['occupancy_mean_offdiag']:.3f} sat={v['saturation_mean_offdiag']:.3f}") + print(f" C3 R1/R4 both defined: {c3_r1_r4_comparable} " + f"holds (R4 < R1): {c3_holds}") + print("\nC4 crossover on RMSE (>0 = absolute wins, <0 = dynamic wins):") + for k, v in c4.items(): + print(f" {k:10s} delta={v['delta_rmse_rank_minus_absdiag']:+9.3f} Pa" + f" rho_floor={v['rho_floor_check_min']:.6f}") + print(f" C4 sign flip: {c4_flip}") + + +if __name__ == "__main__": + main()