hydrate: Dataset als EIN Zip, und VersionedGraph::hydrate_from

.claude/board/EPIPHANIES.md

@@ -1,3 +1,124 @@
+## 2026-08-22 — E-THE-GATE-IS-A-HAND-MAINTAINED-ALLOWLIST-NOT-THE-WORKSPACE-1 — nine workspace members are in no CI job at all, and adding one to `members` adds it to nothing
+
+**Status:** FINDING (measured; every claim is a grep over `.github/workflows/`
+and root `Cargo.toml`). **Confidence:** High. **Supersedes the cause** given in
+`E-A-CRATE-WITH-ZERO-CONSUMERS-IS-BUILT-BY-NOTHING-AND-CAN-BE-MERGED-BROKEN-1`
+(same day, same session, mine).
+
+That entry explained a crate that did not compile at `main` by saying a crate
+with zero consumers is in nobody's build graph. **That is wrong.**
+`crates/lance-graph-hydrate` is listed in `[workspace] members` (root
+`Cargo.toml:25`); `cargo build --workspace` would compile it. The explanation
+was plausible, fitted the symptom, and was never checked against the manifest.
+
+**The actual mechanism.** No workflow in this repo runs `--workspace` or
+`--all`. Every gate names one crate by path:
+
+```
+build.yml:78    cargo build   --manifest-path crates/lance-graph/Cargo.toml
+rust-test.yml   cargo test    --manifest-path crates/<name>/Cargo.toml   (×14)
+style.yml:75+   cargo clippy  --manifest-path crates/<name>/Cargo.toml   (×6)
+style.yml:150+  cargo fmt     --manifest-path crates/<name>/Cargo.toml   (×3)
+```
+
+`style.yml:152` even says so in a comment: *"`cargo fmt --all` never reaches
+it."* The gate is a **hand-maintained allowlist**. Membership in
+`[workspace] members` is therefore not a build guarantee — it is a lockfile and
+`-p` convenience, nothing more.
+
+**Measured scope, not just my crate.** Of 25 members, ELEVEN appear in no
+workflow. Two of those (`lance-graph-catalog`, `lance-graph-planner`) are
+dependencies of `lance-graph`, so their LIBS compile inside a gated build
+(their tests still never run). The remaining NINE are gated by nothing at all:
+
+```
+lance-graph-benches            lance-graph-rbac
+neural-debug                   lance-graph-ontology
+lance-graph-archetype          lance-graph-consumer-conformance
+sigma-tier-router              cognitive-shader-driver
+lance-graph-hydrate
+```
+
+`lance-graph-hydrate` was simply the one that a semver-compatible upstream
+change (`object_store 0.13.2` moving `get`/`put` onto `ObjectStoreExt`) happened
+to break. Nine crates carry the same exposure; nothing about this was specific
+to having no consumer.
+
+**Why the wrong cause was attractive** — worth naming, because it is the
+Kahneman System-1 shape the workspace already warns about: the crate's own
+`lib.rs` prominently says it has zero consumers, so a consumer-shaped
+explanation was *pre-loaded* by the file being read. It explained the symptom
+without contradicting anything visible, which is exactly when a claim needs its
+own check and does not get one. The check was one grep of `Cargo.toml`.
+
+**The durable fix is a gate, not a consumer.** Landing
+`VersionedGraph::hydrate_from` (this branch) does make `lance-graph-hydrate` a
+dependency of a gated crate, so its lib will now compile in CI — but that is a
+side effect, and it leaves its TESTS ungated and the other eight untouched.
+The fix is a workflow line per crate, or one `--workspace` job. Filed as
+`ISSUES.md` `ISS-CI-GATE-IS-AN-ALLOWLIST-NINE-MEMBERS-UNGATED`.
+
+**Outcome, same day (PR #984):** both, and the count above was wrong. A
+workflow line per crate for every member — each measured locally before its
+gate was armed — PLUS `cargo build --workspace` as the net that covers future
+members without a line. Wrong count because the member check extracted with
+`"crates/[a-z0-9-]+"`: no underscore, so `crates/surreal_container` was
+invisible to it, and `tools/dto-class-check` is not under `crates/` at all.
+Eleven, not nine. A membership check blind to two of its inputs is this
+entry's own defect class, one level up, in the instrument rather than the
+workflow — which is the part worth carrying forward.
+
+## 2026-08-22 — E-A-CRATE-WITH-ZERO-CONSUMERS-IS-BUILT-BY-NOTHING-AND-CAN-BE-MERGED-BROKEN-1 — the hydration crate did not compile at `main`, and its own doc says why nobody found out
+
+**Status:** ⊘ SUPERSEDED SAME-DAY by
+`E-THE-GATE-IS-A-HAND-MAINTAINED-ALLOWLIST-NOT-THE-WORKSPACE-1` — the measured
+breakage is real and unchanged; the MECHANISM named below ("zero consumers ⇒
+not in the build graph") is WRONG. `lance-graph-hydrate` is a workspace
+MEMBER. Kept in place per append-only; read the successor for the cause.
+**Confidence:** High on the failure, RETRACTED on the cause.
+
+`crates/lance-graph-hydrate` was minted 2026-08-17 (PR #957 + follow-ups) as
+the generic object-store → local-volume hydration lifecycle, with a 5+3
+hardening council on it. Its `lib.rs` states plainly that it has no consumer
+yet — the deferral of `ISS-HYDRATE-DIR-AND-FILE-DUPLICATE-THEIR-STAGING-BODIES`
+is even justified by that fact (*"cheap only while this crate had zero
+consumers"*).
+
+**Measured 2026-08-22, on a clean checkout of `origin/main` (#981):**
+
+```
+cargo build -p lance-graph-hydrate
+error[E0599]: no method named `get` found for reference `&dyn ObjectStore`
+... 5 errors, in the LIB, not the tests
+```
+
+`object_store 0.13.2` — the version the workspace lockfile already pins, on
+`main`, unchanged by this branch — moved `get`/`put` off the base
+`ObjectStore` trait onto an extension trait `ObjectStoreExt`. The fix is two
+`use` lines. `cargo fmt -p lance-graph-hydrate` also rewrites four of its
+files, so it was merged unformatted as well.
+
+**The finding is not the breakage; it is the mechanism that hid it.** A crate
+with zero consumers is not in any consumer's build graph. Nothing that CI
+actually runs reaches it, so a semver-COMPATIBLE upstream change (0.13.x, no
+major bump, no lockfile movement) silently invalidated it and no gate fired.
+The absence of a consumer was recorded in the crate's own doc as a *cost
+tradeoff* — it is also, and more importantly, a **verification hole**: the
+crate's tests pass only in the one command nobody runs.
+
+Two consequences, both narrow:
+
+1. **A mint without a consumer needs an explicit build gate**, or it is
+   documentation with a `Cargo.toml`. Either land the first consumer in the
+   same arc, or add the crate to whatever CI job actually compiles.
+2. **"Hardened by a 5+3 council" is orthogonal to "builds."** The council
+   reviewed intent, duplication, and doctrine conformance — all real, all
+   preserved by this fix. None of that is a compiler.
+
+Corrected in the same commit that gives the crate its first mechanism with a
+consumer path (`archive::hydrate_archive`), so the hole closes rather than
+being recorded and left open.
+
 ## 2026-08-21 — E-ATTENTION-MASK-IS-A-RENAME-REGISTER-FILE-NOT-A-RESIDUE-CARRIER-1 — the fourth homonym collision of this arc, and the only one where the shipped type is COMPLETE for a different contract
 
 **Status:** FINDING (D-ACR-0, report-only deliverable; every claim a read of

.claude/board/ISSUES.md

@@ -1,5 +1,79 @@
 # Issues Log — Open + Resolved (double-entry, append-only)
 
+## ISS-CI-GATE-IS-AN-ALLOWLIST-NINE-MEMBERS-UNGATED (2026-08-22) — RESOLVED same-day by PR #984
+
+No workflow runs `--workspace` or `--all`. Every gate names one crate by path
+(`--manifest-path crates/<name>/Cargo.toml`), so the CI surface is a
+hand-maintained allowlist and **adding a crate to `[workspace] members` adds it
+to no gate**. `style.yml:152` states this in a comment about `cargo fmt --all`.
+
+Measured 2026-08-22: 11 of 25 members appear in no workflow. Two
+(`lance-graph-catalog`, `lance-graph-planner`) are deps of `lance-graph`, so
+their libs compile inside a gated build but their tests never run. **Nine are
+gated by nothing:** `lance-graph-benches`, `neural-debug`,
+`lance-graph-archetype`, `lance-graph-rbac`, `lance-graph-ontology`,
+`lance-graph-consumer-conformance`, `sigma-tier-router`,
+`cognitive-shader-driver`, `lance-graph-hydrate`.
+
+This is not hypothetical: `lance-graph-hydrate` did not compile at `main`
+(#981) — `object_store 0.13.2`, a semver-COMPATIBLE upstream release already
+in the lockfile, moved `get`/`put` onto an extension trait. It was also
+fmt-dirty and carried a live clippy warning. Full account:
+`EPIPHANIES.md` `E-THE-GATE-IS-A-HAND-MAINTAINED-ALLOWLIST-NOT-THE-WORKSPACE-1`.
+
+**Fixed here:** `lance-graph-hydrate` gets its `rust-test.yml` + `style.yml`
+lines in this branch, and becomes a dependency of `lance-graph` (so its lib
+also compiles inside an existing gate).
+
+**RESOLVED 2026-08-22 by PR #984**, and the resolution went further than what
+this entry proposed. The text below is kept as written — it was the state when
+filed — with the outcome recorded here rather than edited into it:
+
+- All members are gated now, not just `lance-graph-hydrate`. #984 measured each
+  crate locally first and added a test step plus a rustfmt step per crate.
+- `cargo build --workspace` landed as the structural net, so a FUTURE member is
+  covered the moment it is listed in `[workspace]`. That was the standing
+  choice this entry left to the operator; it was taken.
+- The count in this entry's own title is WRONG and stays wrong on purpose. It
+  says nine; there were eleven, because the member check behind it extracted
+  with `"crates/[a-z0-9-]+"` — no underscore — and could not see
+  `crates/surreal_container` or `tools/dto-class-check`. A membership check
+  blind to two of its inputs is the same defect class this entry describes,
+  one level up. Both are gated in #984.
+- Not taken, still open: a `cargo test --workspace` job. Measured at 14 GB
+  across 86 binaries against 3.5 GB for the compile — the same order as a
+  runner's free disk. The per-crate test steps remain, so a new member's TESTS
+  still need a line.
+
+Original text, as filed:
+
+**Still open — deliberately not fixed blind:** the other eight. Some exclusions
+may be intentional (a benches crate, a research crate), and adding eight jobs
+without knowing which are meant to be gated would trade a silent hole for
+silent CI cost. What is needed is one decision — either a single `--workspace`
+job (cheapest, catches every future member automatically) or an explicit,
+per-crate rationale for each omission recorded next to the allowlist. That
+decision is the operator's; this entry is the measurement it needs.
+
+
+## ISS-HYDRATE-CRATE-HAS-NO-BUILD-GATE (2026-08-22) — SUPERSEDED same-day by ISS-CI-GATE-IS-AN-ALLOWLIST-NINE-MEMBERS-UNGATED (the cause was mis-stated; the crate is a workspace MEMBER)
+
+`crates/lance-graph-hydrate` did not compile at `origin/main` (#981) and was
+also fmt-dirty; both are fixed in the commit that files this. See
+`EPIPHANIES.md`
+`E-A-CRATE-WITH-ZERO-CONSUMERS-IS-BUILT-BY-NOTHING-AND-CAN-BE-MERGED-BROKEN-1`
+for the measurement and the mechanism.
+
+**What is fixed:** the two `ObjectStoreExt` imports and the formatting.
+
+**What is NOT fixed, and is the actual issue:** nothing in CI compiles this
+crate, because no consumer depends on it. The same class of upstream change
+can invalidate it again tomorrow with no gate firing. The durable fix is one
+of: (a) land the first real consumer — `VersionedGraph::hydrate_from`, the
+piece `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE` names — so
+the crate enters `lance-graph`'s build graph; or (b) add it explicitly to the
+workspace job that runs on every PR. (a) is preferable: it closes two issues
+with one edge instead of adding a gate around an unused artifact.
 ## ISS-CAUSAL-EDGE-CARRIES-SEVEN-PRE-EXISTING-CLIPPY-FINDINGS (2026-08-22) — OPEN
 
 `crates/causal-edge` is workspace-EXCLUDED but a path-dep of `lance-graph`,

.claude/board/LATEST_STATE.md

@@ -1,5 +1,70 @@
 ## 2026-08-21 — D-ACR-7 IMPLEMENTED — `contract::band_reading` (the 59..63 reading contract)
 
+### Current Contract Inventory — `VersionedGraph::hydrate_from`, and the CI gate that should have caught all of this
+
+- **`lance_graph::graph::versioned::VersionedGraph::hydrate_from`** (new) —
+  the shape `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE` names.
+  `local`/`s3`/`azure`/`gcs` address a store WHERE IT SITS; for the three
+  remote ones that makes the object store the store, which the doctrine
+  explicitly does not. They are KEPT (addressing a remote store is legitimate
+  when a caller means it, and better at >1 replica); this adds the doctrine's
+  own shape beside them.
+  - **Ensure-hydrated, not hydrate-or-fail**: an existing destination is the
+    warm path, returned as `Hydration::AlreadyLocal` rather than an error.
+  - **`Hydration { Fresh(ArchiveReport), AlreadyLocal }`** (new) — the
+    distinction is returned rather than folded away: a boot that fetched and a
+    boot that found it are different events.
+  - **`GraphError::Hydration { source, location }`** (new variant) — its own
+    variant, not a flattened message, so a caller can tell a checksum mismatch
+    (never retry) from a transport error (retry).
+- **CI is no longer this branch's concern — PR #984 carries it.** An earlier
+  draft of this entry said `lance-graph-hydrate` is "gated for the first time"
+  here and that "eight more members are still ungated". Both were true when
+  written and are false now: #984 gated EVERY member (each measured locally
+  first), added `cargo build --workspace` as the structural net so future
+  members need no line, and pinned the toolchain in all seven workflows. The
+  count was also wrong — eleven, not nine, because the member check behind it
+  could not see names with an underscore. See `ISSUES.md`
+  `ISS-CI-GATE-IS-AN-ALLOWLIST-NINE-MEMBERS-UNGATED` (now RESOLVED, with the
+  outcome recorded above its original text) and `EPIPHANIES.md`
+  `E-THE-GATE-IS-A-HAND-MAINTAINED-ALLOWLIST-NOT-THE-WORKSPACE-1`.
+- Tests: `lance-graph-hydrate` 39 green; `lance-graph` `--lib` green incl. two
+  new `hydrate_from` falsifiers scoped to what that function adds (the warm/
+  error mapping — the archive mechanics are falsified one crate down).
+  `cargo clippy -p lance-graph --lib --tests -- -D warnings` clean.
+
+### Current Contract Inventory — 1 new module in `lance-graph-hydrate`, and that crate now COMPILES
+
+- **`lance_graph_hydrate::archive`** (new) — the `absent -> hydrated` edge for a
+  dataset shipped as ONE checksum-pinned **zip** object, alongside the existing
+  `copy::hydrate_dir` (a tree of objects) and `file::hydrate_file` (one plain
+  object).
+  - `hydrate_archive(store, remote_object, publish_dir, expected_sha256_hex,
+    root) -> ArchiveReport` — composes the two mechanisms the crate already
+    owns (`hydrate_file` for the pinned fetch, `publish::publish_by_rename` +
+    `StagingKind::Dir` for the atomic publish) and adds only the middle:
+    expanding one verified container into staging under a containment rule.
+  - `ArchiveReport { files, bytes }`; `HydrateArchiveError` adds
+    `EscapingEntry { entry, root }` (Zip-Slip refusal, whole archive rejected
+    off the CENTRAL DIRECTORY before any byte is written) and
+    `NoFiles { root }` (an all-directory tree would publish an unopenable
+    dataset).
+  - **Zip and not tar, by operator ruling 2026-08-22** (*"bitte als zip, nicht
+    dass wir ein Verzeichnis mit einzelnen Dateien shippen"*): a zip carries a
+    central directory, so entries can be enumerated and sought without a
+    sequential scan — which is also what makes the whole-index validation above
+    possible before extraction starts.
+- **`lance-graph-hydrate` builds again.** It did not compile at `origin/main`
+  (#981): `object_store 0.13.2` moved `get`/`put` onto `ObjectStoreExt`. Two
+  `use` lines; four files also picked up `cargo fmt`. Why a merged crate could
+  be broken at HEAD — and what that says about minting without a consumer — is
+  `EPIPHANIES.md`
+  `E-A-CRATE-WITH-ZERO-CONSUMERS-IS-BUILT-BY-NOTHING-AND-CAN-BE-MERGED-BROKEN-1`;
+  the durable gap is `ISSUES.md` `ISS-HYDRATE-CRATE-HAS-NO-BUILD-GATE`.
+- Tests: 39 green in the crate (33 pre-existing, 6 new). Both new guards are
+  mutation-checked — disabling the containment check and the file counter turns
+  exactly their two tests red and leaves the other four green.
+
 ### Current Contract Inventory — 1 new zero-dep module + 1 ClassView provided method + 1 gate test in `causal-edge`
 
 - **`lance_graph_contract::band_reading`** (new, zero new bytes) — implements