diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 252a15fda..dcdcc8408 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,69 @@ +## 2026-08-09 + +### E-THE-ARTIFACT-WRITE-DECIDES-WHAT-KANBAN-PROGRESS-BECOMES-DURABLE-1 + +**FINDING (operator-ruled, implemented Phase A).** The persistence question was +being asked backwards. The old contract let the *cycle* decide when to write — +every 550 ms sweep sealed a `DatasetVersion`, so a thought crossing dozens of +cheap Rubicon steps in ~2 s minted dozens of versions and stretched a ~2 s +ladder across ~32-35 s of barriers. The correct rule inverts the initiative: + +> **Kanban never decides when to persist. The semantic write that happens +> anyway decides which kanban progress gets a durable anchor.** + +Thinking is cheaper than persisting its intermediate control flow. Rubicon +state, `Continue`, `Hold`, held work, the current rung and scheduler progress +stay **transient** — after a crash they are cheaply regenerated from the pinned +sealed inputs, which is exactly the property that makes discarding them safe. + +**The mechanical form is smaller than the ruling sounds.** The gate already +existed in the data: a cast's payload. Non-empty = an artifact (grounding +result, reusable walk product, adjudication, conclusion) → persisted. +Empty = intent-only → ephemeral. `restage_held` had been casting empty payloads +since #879 (`cycle_driver.rs:303`), and #911's 512-byte ABI gate contradicted +it — the *fix* was not padding those casts to 512 bytes to satisfy the gate but +recognizing that **the empty payload IS the ephemerality mechanism**. Two +post-merge P1s dissolved at once: intent-only casts can never trip a payload +gate they never reach, and the empty-cycle version disappears because zero +artifact casts means the sink is never called at all. + +**The general lesson.** When a gate and a producer contradict each other, the +question "which one do I bend?" often has a third answer: the contradiction is +the system telling you the two things were never in the same category. A +lifecycle transition and a semantic artifact are not the same kind of fact, and +only one of them belongs in durable storage. + +### E-A-PUBLISHED-MANIFEST-IS-HISTORY-RECONCILIATION-NOT-ROLLBACK-1 + +**FINDING (measured against `lance-9.0.0/src/io/commit.rs:914-950`).** #911 +tried to make an optimistic fence *effective* after the fact: on detecting that +a foreign writer had shifted the published version, it issued a compensating +`Dataset::delete` scoped to the cycle, then returned a retryable error. Review +found the flaw (the `(cycle, base_version)` predicate can delete a *successful +concurrent writer's* rows — the very race it exists to handle), but the deeper +error is categorical: **`Dataset::delete` creates another version. It is not +rollback.** There is no undo in an MVCC manifest chain. + +The measured constraint underneath: **Lance 9 has no atomic expected-version +fence for `Append`** — the conflict rebase runs even on a single-attempt commit; +strict no-rebase mode exists only for `Overwrite`. A read-then-append is not a +compare-and-swap, and dressing it as one produces exactly the +committed-but-reported-failed hole that made the driver regenerate work that +had already landed. + +The resolution is to stop trying to make the fence retroactive and make +**reconciliation authoritative** instead: commit the batch's identity +`(cycle, batch_hash)` *in the same commit as its rows*, look it up before +appending, and let a lost acknowledgement resolve by re-submitting the SAME +frozen batch. Same identity + same hash ⇒ `Reconciled` (success, no second +append). Same identity + different hash ⇒ fail closed. Genuinely unknown ⇒ an +`Ambiguous` state that says so, instead of an error that falsely promises +nothing landed. + +**The rule this leaves behind:** never return an error meaning "nothing +landed" when failure could have occurred after publication — and never delete +history to restore an expected version number. + ## 2026-08-07 ### E-COMPRESSION-META-INERT-AT-512-STRIDE-1 diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index f163e3068..0f7e25203 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,14 @@ +## 2026-08-09 — persistence-artifact-backed-commit v1 — RATIFIED (Phase A implemented; B–F planned) — main thread + +The canonical persistence contract, replacing the #911 cycle-persistence model. +Plan: `.claude/plans/persistence-artifact-backed-commit-v1.md`. + +- **The rule:** no artifact-backed semantic change → no write → no new `DatasetVersion`. Thinking runs its whole Rubicon ladder transiently; only a semantic artifact becomes durable, and kanban progress rides along in the commit that was happening anyway. +- **One logical writer**, split by capability: `Clone + &self` for producer submission / read-only projections; the concrete `LanceCycleWriter` non-`Clone`, owning its `Dataset` handle + head, committing through `&mut self`. +- **No rollback, no compensating delete** — a published manifest is history (measured: Lance 9 has no atomic expected-version Append fence); durable in-band `(cycle, batch_hash)` makes reconciliation authoritative. +- **Zero reload on the normal path**, instrumented by `opens()`; reads bounded (`after_cycle`) and projected (timeline never touches payloads). +- **Phases:** A (this contract + owned writer) — implemented on `claude/phase-a-owned-writer`; B shared representation/projection ABI; C live granular visibility + wavefront; D conclusion boundary; E MedCare proof + Gotham wiring; F A2UI/ClassView renderer. Each phase restarts from merged main; public lance-graph stays generic, clinical mappings stay private to MedCare-rs. + ## 2026-08-06 — idle-flush-dataset-eviction v1 — PROPOSAL (not scheduled; nothing implemented, nothing measured) **Plan:** `.claude/plans/idle-flush-dataset-eviction-v1.md` diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 24a814745..34b336a12 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,16 @@ +## 2026-08-09 — branch `claude/phase-a-owned-writer` — Phase A: the artifact-backed commit contract + the SOLE owned Lance writer (`LanceCycleWriter`) + +> **⊘ This entry SUPERSEDES the #911 entry below it** (operator ruling 2026-08-09). The §I.6 "every cycle publishes exactly one `DatasetVersion`" contract, the compensating delete, and the per-operation-reopen sink are all REMOVED — not repaired. Canonical record: `.claude/plans/persistence-artifact-backed-commit-v1.md`. Nothing below is deleted; it is read through that document. + +### Current Contract Inventory — reshaped (lance-graph-planner) + replaced (lance-graph core, `planner` feature) + +- **The governing storage rule** (`persist_sink`): **no artifact-backed semantic change → no write → no new `DatasetVersion`.** Thinking is cheaper than persisting its intermediate control flow: a thought runs its whole Rubicon ladder transiently and only an artifact-backed delta becomes durable. Mechanically the gate is the cast payload — NON-EMPTY = artifact cast (persisted), EMPTY = intent-only cast (held-intent re-stage / pure kanban step) which `persist_cycle` partitions out as **ephemeral**. Zero artifact casts ⇒ `CommitOutcome::NoChange { head }` with the sink NEVER called (zero store ops, zero rows, unchanged version). `restage_held`'s empty payload is thereby exactly what makes a re-staged intent ephemeral — no ABI gate can trip on it, because such a cast never reaches the writer. #911's deliberate empty-cycle versioning is REMOVED and its falsifier inverted. +- **`WalSink` reshaped**: `commit_cycle(&mut self, batch) -> Result` (the `base` param is gone — it lives in `batch.frame`), `scan_sealed(after_cycle: Option)` (cycle-bounded tail, pushed into the scan), `timeline() -> Vec` (replaces `versions()`; frame metadata only, never a payload). `LandedSlot` is now `{ cycle, slot }` — the derived `version` field is GONE (a physical publication position is not a per-row semantic identity). `DetachedCycleBatch` gained `batch_hash` (FNV-1a 64 over the CANONICAL content, so randomized completion order yields an identical hash). +- **Honest commit states**: `NoChange { head }` / `Committed { version, cycle, batch_hash }` / `Reconciled { version, cycle, batch_hash }`; errors `Fenced { current_head }` / `HashConflict { cycle, stored_hash, offered_hash }` / `Io(WriteFailed)` / `Ambiguous { cycle, batch_hash, cause }`. **No error may promise "nothing landed" when failure could have occurred after manifest publication** — that case reconciles or surfaces as `Ambiguous`. +- **`LanceCycleWriter`** (replaces `LanceCycleSink`) — the SOLE application writer: **non-`Clone`**, owns a **long-lived `Dataset` handle + in-memory head**, commits through **`&mut self`**. The capability split is deliberate (operator correction): `Clone + &self` belongs to producer submission and read-only projections; fire-and-forget means producers get no acknowledgement, NOT that the writer ignores the result. **No rollback, no compensating delete** — a published manifest is history; `Dataset::delete` mints another version and is not rollback (and #911's `(cycle, base_version)` predicate could destroy a concurrent same-cycle winner). Idempotency is durable and in-band: `(cycle, batch_hash)` committed with the rows, reconciled FIRST, so re-submitting the same frozen batch after a lost acknowledgement returns `Reconciled` instead of double-appending. Lance 9 has NO atomic expected-version fence for Append (rebase runs even single-attempt; strict mode is Overwrite-only — measured in `lance-9.0.0/src/io/commit.rs:914-950`); that is stated, not papered over. +- **Zero reload on the normal path, instrumented**: `LanceCycleWriter::opens()` counts every `Dataset::open` ever performed (startup + ambiguity resolution only). Layout is three row kinds — frame (1/cycle), landing metadata (1/artifact cast, payload NULL), coalesced image (1 per DIRTY ROW, the final 512-byte payload). Payload is physically `FixedSizeBinary(512)`; reads are bounded + projected (`timeline` never scans the payload column; `scan_sealed` returns transition metadata only; `scan_image` projects payload on request). +- **Gates:** 11 reopened-store falsifiers in `cycle_sink.rs` + 5 contract falsifiers in `persist_sink.rs`, including the **measured bytes-written** one — 64 transient breaths on one row cost **512 durable bytes, not 64 × 512**. Honestly deferred (named, not skipped): the real object-store RUN — S3 needs lance's `aws` feature, which our `lance = "=9.0.0"` default-features pin already enables (verified: `aws-config`/`aws-credential-types` in the graph), so an `s3://` store compiles and routes today; what is unmeasured is the credentialed commit/reconciliation/tail-read, and **no object-store durability claim is made** until it runs. Also deferred: `Continue`-from-pulse (Phase C), 64k-scale measurement. + ## 2026-08-09 — branch `claude/medcare-rs-continue-ufsazd` — `lance_graph::graph::cycle_sink`: the CONCRETE cognitive-cycle Lance sink (the storage-proven `WalSink`) ### Current Contract Inventory — new module (lance-graph core, `planner` feature) diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 366be30e3..f01d73920 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -33,6 +33,16 @@ > - **Docs** — knowledge files produced (immutable) > - **Confidence (YYYY-MM-DD):** — the ONLY mutable field +## 2026-08-09 — branch `claude/phase-a-owned-writer` (PR pending) — Phase A: the artifact-backed commit contract + the SOLE owned writer + +- **Added.** `.claude/plans/persistence-artifact-backed-commit-v1.md` (the canonical persistence contract, ratified). `lance_graph::graph::cycle_sink::LanceCycleWriter` (~900 LOC incl. 11 reopened-store falsifiers) replacing `LanceCycleSink`. In `persist_sink`: `CommitOutcome` / `CommitError` / `FrameMeta`, `DetachedCycleBatch::batch_hash`, the reshaped `WalSink` (`commit_cycle(&mut self, batch)`, `scan_sealed(after_cycle)`, `timeline()`), `PersistError::Commit`, and 5 contract falsifiers. +- **Locked.** **No artifact-backed semantic change → no write → no new DatasetVersion** (empty/intent-only cycles perform ZERO store operations; `restage_held`'s empty payload IS the ephemerality mechanism). **One logical writer, split by CAPABILITY** — `Clone + &self` for producer submission and read-only projections; the concrete writer non-`Clone`, owning its `Dataset` handle + head, committing through `&mut self`; fire-and-forget means producers get no acknowledgement, never that the writer ignores the result. **No rollback and no compensating delete** — a published manifest is history; idempotency is durable in-band `(cycle, batch_hash)`, reconciled first, so a lost acknowledgement is resolved by re-submitting the same frozen batch. **No error may promise "nothing landed" after possible publication** (`Ambiguous` is the honest unknown). **Zero reload on the normal path**, instrumented via `opens()`. **Lance 9 has no atomic expected-version Append fence** — measured at `lance-9.0.0/src/io/commit.rs:914-950`, stated rather than papered over. **`DatasetVersion` is a physical publication position**, not a per-row semantic identity (`LandedSlot.version` removed). +- **Deferred.** Real object-store commit + ambiguous-response paths (no credentials in this environment — NO object-store durability claim is made). True zero-copy: the copy boundary (Arrow builder materialization + `to_vec` readback) is documented and isolated for a later measured PR, along with the `BatchWriter

`-descriptor-vs-`Vec` contradiction. Phases B–F (representation/projection ABI, live pulse + wavefront, conclusion boundary, MedCare proof, A2UI renderer). +- **Docs.** The canonical plan above; a `⊘ PARTIALLY SUPERSEDED` header on `persistence-cycle-wal-bootstrap-v1.md` (guarantee 5 now conditional; guarantees 1–4/6 survive, §2 sparse-delta now IMPLEMENTED); LATEST_STATE entry marking the #911 entry superseded. + +**Confidence (2026-08-09):** contract + writer falsifiers green against reopened local stores; not yet merged; object-store unproven by design. +**Correction (2026-08-09, review round on #912):** five review findings fixed in place — (1) `reopen` can no longer degrade an existing store to empty (a held handle survives a transient NotFound; outcome stays Ambiguous); (2) the in-process one-writer topology is now ENFORCED via a process-local path registry (second live `open` refused; Drop frees), cross-process stays a documented deployment lease; (3) the normal commit path is now genuinely scan-free (cycle watermark seeded at open; `reconcile_scans()` instrumented and falsified at 0 for fresh monotonic commits; reconciliation only on fence-fail / re-submission / ambiguity); (4) `Reconciled.version` renamed `current_head` — it is NOT a publication version; only `Committed.version` is audit-grade; (5) `recover_fleet` now reports `foreign_landings` + `foreign_min_cycle` (the latecomer fence: never raise the global bound past it). Honestly recorded, not fixed: the artifact gate tests payload PRESENCE not semantic CHANGE (Phase-D conclusion-identity refinement, documented in the module doc); the 64-breaths falsifier now states the 64 compact landing-metadata rows explicitly (per-cast metadata collapse = the Phase B/C KanbanRollup); `run_cycle`'s doc no longer claims the fleet borrow is released across the await — it names the split (`collect → seal_cycle → apply`) as the production detached path. + ## 2026-08-09 — branch `claude/medcare-rs-continue-ufsazd` (PR pending) — the concrete cognitive-cycle Lance sink: `graph::cycle_sink::LanceCycleSink` - **Added.** `lance_graph::graph::cycle_sink` (~660 LOC incl. 6 reopened-dataset tokio tests) — the concrete `lance_graph_planner::persist_sink::WalSink` over the official Lance 9 insert path; `cycle_store_schema()` (frame row + landing rows, nullable Rubicon `move_*` columns, `payload` witness bytes); `LanceCycleSink`. Module gated on the default-on `planner` feature. @@ -40,7 +50,7 @@ - **Deferred.** The MedCare consumer arc (production `drive_cohort_thoughts` caller, witness-seal, views reading the sealed version) — next PR, in MedCare-rs. `recover_and_apply` wiring against this sink in a production driver. Object-store (s3/az/gs) smoke — the path plumbing accepts URIs but only local was exercised. - **Docs.** Module-level witness/§I.6 contract in `cycle_sink.rs`; this entry + LATEST_STATE inventory (same commit). -**Confidence (2026-08-09):** tests green against reopened local datasets; not yet merged. +**Confidence (2026-08-09):** MERGED as `8a5be50`. **SUPERSEDED the same day by the Phase-A entry above** (operator ruling): the §I.6 one-version-per-cycle contract, the compensating delete, and the per-operation reopen are removed — the post-merge review also confirmed two P1s in this entry's shape (the delete's `(cycle, base_version)` predicate can destroy a concurrent same-cycle winner; the 512-byte gate contradicts `restage_held`'s intent-only empty payload). Both are resolved structurally in Phase A rather than patched here. ## 2026-08-05 — the lance 9 / DataFusion 54 / Rust 1.97.1 cross-repo bump (9 repos; lance-graph PR pending, siblings MERGED) diff --git a/.claude/plans/persistence-artifact-backed-commit-v1.md b/.claude/plans/persistence-artifact-backed-commit-v1.md new file mode 100644 index 000000000..ece1359da --- /dev/null +++ b/.claude/plans/persistence-artifact-backed-commit-v1.md @@ -0,0 +1,334 @@ +# persistence-artifact-backed-commit-v1 — the canonical persistence contract + +> **Status:** RATIFIED (operator ruling 2026-08-09). Phase A **implemented** on +> branch `claude/phase-a-owned-writer`; Phases B–F planned (§7). +> **Supersedes:** the cycle-persistence contract as shipped in PR #911 and the +> "one version per cycle, empty cycles included" reading of +> `persistence-cycle-wal-bootstrap-v1.md` §1 guarantee 5. Those documents are +> NOT deleted — they are read through this one, which is authoritative wherever +> they disagree. +> **Owns:** what may become durable, who may write it, what a commit returns, +> and what the normal path may read. +> **Does NOT own:** the OGAR-loco representation (Phase B), granular kanban +> visibility (Phase C), the clinical conclusion boundary (Phase D), MedCare +> wiring (Phase E), the A2UI renderer (Phase F). + +--- + +## 1. The governing storage rule + +**No artifact-backed semantic change → no write → no new `DatasetVersion`.** + +Thinking is cheaper than persisting its intermediate control flow. A thought +runs its complete Rubicon/Heckhausen ladder **transiently** — often dozens of +cheap steps — and only what survived the ladder becomes durable: + +```text +sealed inputs + → complete transient thought ladder + → artifact-backed semantic delta + → canonical ABI row + compact metadata + → non-empty batch + → ONE official Lance MVCC commit +``` + +NOT `Planning → persist → CognitiveWork → persist → Continue → persist → …`. + +A commit is allowed only for a real semantic delta: a new or changed grounding +result, a genuinely reusable ontology/RO walk product, an adjudication, a +terminal conclusion revision, a non-repeatable external-effect receipt, or +another explicitly registered semantic artifact. + +**A timer tick, an empty cycle, `Continue`, `Hold`, a changed live priority, a +census observation, scheduler movement or a granular phase pulse is NOT a +durable artifact.** + +Kanban never decides when to persist. If a semantic artifact is being written +anyway, the kanban progress accumulated since the preceding artifact rides +along as compact metadata in that same commit, under the same idempotency +identity. + +### The mechanical form (Phase A, implemented) + +The artifact gate is the cast's payload: + +| cast shape | meaning | fate | +|---|---|---| +| NON-EMPTY payload | artifact cast — a real semantic delta | persisted | +| EMPTY payload | intent-only cast (held-intent re-stage, pure kanban step) | EPHEMERAL — never reaches the store | + +`persist_cycle` partitions intent-only casts out **before** the freeze, so: + +```text +zero artifact casts + → CommitOutcome::NoChange { head } + → zero sink calls · zero rows · zero frames · unchanged DatasetVersion +``` + +`restage_held`'s empty payload is therefore not a defect to be padded around — +it is precisely what makes a re-staged intent ephemeral. No payload/ABI gate +can ever trip on it, because such a cast never reaches the writer. + +**#911's deliberate empty-cycle versioning is REMOVED, not repaired.** Its +falsifier `empty_cycle_advances_timeline_only` is inverted into +`no_artifact_delta_writes_nothing_and_creates_no_version`. + +--- + +## 2. One logical writer — capability split, not merely method mutability + +There is exactly **one logical application writer** per cycle store. The 64k +thoughts / SoA owners are parallel **producers**, not Lance writers: + +```text +independent SoA owners/thoughts + → cast_on_behalf(owner) (fire-and-forget: no acknowledgement) + → ephemeral BatchWriter staging + → deterministic collect/freeze + → ONE detached cycle batch + → the SOLE writer's &mut commit + → returned DatasetVersion becomes the new head +``` + +The split is by **capability**, not by uniform mutability (operator +correction, 2026-08-09): + +| surface | shape | why | +|---|---|---| +| producer submission + read-only projections | `Clone + &self` | fire-and-forget; many producers | +| the concrete Lance writer | **non-`Clone`**, owns the `Dataset` handle + head, commits through **`&mut self`** | two application commits cannot interleave through the type boundary | + +**Fire-and-forget means producers receive no acknowledgement — it does NOT mean +the sole writer ignores the commit result.** The writer fully honors every +outcome (§3). + +Lance's own transaction/manifest machinery (the backend durability path) is +**internal to that one writer**; `Dataset::write` / `Dataset::append` are +official atomic Lance MVCC commits. Deliberately absent: per-plan writer +leases, multi-writer consensus, a second WAL, a bespoke confirmation ledger, +64k actor acknowledgements, a foreign-writer recovery protocol. + +An unexpected head can therefore mean only: an earlier commit became durable +but its response was lost; a restart reopened from a stale cached head; +unauthorized maintenance or another writer violated the topology; or +corruption. **It is a fence/reconciliation condition, never normal +competition.** + +--- + +## 3. No rollback, no compensating delete — reconciliation is authoritative + +**Measured, not assumed:** Lance 9 has **no atomic expected-version fence for +Append`. The conflict rebase runs even on a single-attempt commit; strict +no-rebase mode exists only for `Overwrite` +(`lance-9.0.0/src/io/commit.rs:914-950`). This is stated honestly rather than +papered over with a read-check pretending to be compare-and-swap. + +Once Lance publishes a manifest, that commit is **history**. `Dataset::delete` +creates *another* version and is not rollback — **#911's compensating delete is +removed entirely, not repaired with a nonce** (its `(cycle, base_version)` +predicate could also destroy a concurrent same-cycle winner's rows). + +Instead, idempotency is **durable and in-band**: every committed batch carries +its `(cycle, batch_hash)` in the same commit, and the writer **reconciles +first**. + +```rust +CommitOutcome::NoChange { head } // nothing to write +CommitOutcome::Committed { version, cycle, batch_hash } // durable now +CommitOutcome::Reconciled { version, cycle, batch_hash } // was already durable +CommitError::Fenced { current_head } // nothing written +CommitError::HashConflict { cycle, stored_hash, offered_hash } // fail closed +CommitError::Io (WriteFailed) // nothing published +CommitError::Ambiguous { cycle, batch_hash, cause } // genuinely unknown +``` + +The rules that bind them: + +- Failure **proven** before publication → discard transient staging, regenerate + from the unchanged `Vn`. +- Success → accept the **actual returned version**; never "correct" it. +- Timeout / lost response → **re-submit the SAME frozen batch**; the writer's + reconciliation-first lookup returns `Reconciled`, so the retry cannot + double-append. +- Same identity, different hash → **fail closed**; never promote, never + overwrite. +- Never delete history to restore an expected version number. +- Never regenerate assuming nothing landed until reconciliation proves it. + +**No generic error may promise "nothing landed" when failure could have +occurred after manifest publication.** `Ambiguous` is the honest "I don't +know", and it is only reachable when reconciliation itself could not answer. + +`batch_hash` is FNV-1a 64 over the **canonical** (already deinterlaced + +coalesced) content, so randomized worker completion order yields an identical +hash: determinism comes from canonical content, never from arrival order or a +process-local counter. + +--- + +## 4. Reference the new version; never reload normal state + +After a successful commit the caller **already knows** the submitted batch, the +affected owners and rows, the sparse transitions, the batch hash, and the +returned version. The hot path is therefore: + +```text +outcome = commit(batch) +current_head = outcome.version +apply_sparse_effects_from_the_submitted_batch() +continue +``` + +It must NOT reopen the dataset, call the timeline, scan sealed landings, scan +the image, recover the fleet, rehydrate SoAs, replay the timeline, or run +read-time deinterlace to rediscover its own write. + +**Instrumented, not asserted:** `LanceCycleWriter::opens()` counts every +`Dataset::open` the writer has ever performed. The falsifier +`successful_commit_reopens_nothing` drives three commits and asserts the count +stays at its post-startup value. The writer holds ONE long-lived `Dataset` +handle plus an in-memory head token; it re-opens only at startup and to resolve +an ambiguous outcome (both counted). + +Reads are bounded and projected: + +| read | bound | projection | +|---|---|---| +| `timeline()` | frame rows only (`kind = 0`) | `cycle`, `base_version`, `batch_hash` — payload column never scanned | +| `scan_sealed(after_cycle)` | `cycle > bound`, pushed into the Lance scan | transition metadata only; landing rows carry NO payload | +| `scan_image(cycle)` | `kind = 2 AND cycle = …` | `row` + payload, on request only | + +Recovery takes an explicit `after_cycle` bound — the unbounded +`scan_sealed(None)` full-history read is no longer the recovery path. + +--- + +## 5. Physical layout (Phase A) + +One dataset, three row kinds, payload physically `FixedSizeBinary(512)` +(nullable — only image rows carry it): + +| kind | rows per cycle | carries | +|---|---|---| +| 0 frame | 1 | cycle · base_version · batch_hash | +| 1 landing | one per artifact cast | stream_position · owner · row · move_* (nullable) · payload NULL | +| 2 image | one per DIRTY ROW | row · the FINAL 512-byte payload after the fold | + +**The measured consequence (falsifier +`sixty_four_breaths_on_one_row_cost_one_image_row`):** 64 successive artifact +updates to one row produce **512 durable payload bytes**, not 64 × 512. The +coalesced image is the durable end-form; the per-cast landing rows are compact +transition metadata that recovery needs and carry no payload at all. + +The 512-byte witness ABI is enforced **physically** (the Arrow column type) and +**at build time** (an artifact payload of any other length refuses the whole +commit before anything durable happens) — and it can only ever apply to +artifact casts, since intent-only casts never reach the writer. + +### The honest copy boundary + +Phase A materializes the frozen batch's payloads into Arrow builders (one copy +each) and copies bytes back out on read (`to_vec`). True zero-copy (Arc-backed +Arrow buffers pinned over SoA ranges) does **not** fit this focused repair. The +copy boundary is exactly those two seams and nothing else; it is isolated for a +later measured PR rather than claimed away. `BatchWriter

`'s doc calls `P` a +descriptor while `cycle_driver` instantiates `BatchWriter>` — that +contradiction is recorded here, not silently corrected. + +--- + +## 6. `temporal.rs` — oracle, not ordering service + +`temporal.rs` remains the temporal **admissibility and replay oracle**: pick a +pinned horizon, prove no future knowledge entered a result, reproduce +deterministic thought, validate latecomers and cohort composition, support +audit and restart recovery. + +It is **not** a global write-order weaver, not the scheduler, not kanban +authority, not a normal post-commit operation, and never a reason to reload the +current fleet. The workspace's measured finding stands unchanged +(`.claude/knowledge/seal-vs-temporal-ordering-information.md`): the seal +computes a cross-owner total order, arrival as a durable input, the per-row +destructive fold, and the cohort boundary — none of which the per-owner +temporal projection encodes. + +`DatasetVersion` remains a **physical publication position**. Combined with +dataset identity and an anchor identity it is also the simplest durable audit +address; the version alone carries no semantic identity. + +--- + +## 7. Phase sequence + +| phase | scope | state | +|---|---|---| +| **A** | this contract + the owned `LanceCycleWriter` | **implemented** (branch `claude/phase-a-owned-writer`) | +| B | shared representation + projection ABI (`VersionRef`, `PhasePulse`, `KanbanRollup`, artifact-backed `DurableAnchor`, OGAR-loco phase mapping, pure KanbanView/GothamProjection + golden rebuild) | planned | +| C | live granular visibility + wavefront (`EphemeralProgress`, `Continue`, pulse coalescing, plateau prioritization, latecomer semantics, rs-graph-llm/Blockly seams — zero durable pulse writes) | planned | +| D | conclusion boundary (`ConclusionRevision` as terminal `DurableAnchor`, immutable `(patient_episode, plan_id, revision)` identity, predecessor `VersionRef`, witness root, cohort manifest) | planned | +| E | MedCare proof + Gotham wiring (production caller, one real persisted medical thought, restart/reopen, bounded sealed read, no request-time reconstruction presented as sealed) | planned | +| F | A2UI/ClassView renderer (ABI projection, foveated ontology hydration, PII-free overlays, widefield masks) | planned | + +Each phase restarts from merged `main`; public lance-graph stays generic and +clinical mappings stay private to MedCare-rs. + +--- + +## 8. Authority table + +```text +OGAR-loco Representation plan truth (Phase B) +PhasePulse / EphemeralProgress granular live, crash-lossable visibility (Phase C) +KanbanRollup + semantic artifact compact durable progress/result truth +VersionRef Lance-native audit address +KanbanView / GothamProjection derived views over the same source +WitnessArcFacet evidence / ontology / crosswalk detail +temporal.rs horizon / replay / admissibility oracle +Lance DatasetVersion physical publication position +Lance commit machinery backend durability of the SOLE writer +``` + +--- + +## 9. Phase-A falsifiers (implemented, green) + +`crates/lance-graph/src/graph/cycle_sink.rs` (11, every one against a REOPENED +store — a fresh `LanceCycleWriter::open` over the same path): + +1. `no_artifact_delta_writes_nothing_and_creates_no_version` — 2,000 intent-only casts ⇒ `NoChange`, no dataset created, restart still empty, then a real cycle commits at V1. +2. `sixty_four_breaths_on_one_row_cost_one_image_row` — the measured bytes-written falsifier (512 B, not 64 × 512). +3. `successful_commit_reopens_nothing` — `opens()` flat across three commits. +4. `bounded_tail_recovery_reads_no_payloads` — `after_cycle` bound + payload column never projected, transition metadata intact. +5. `timeline_is_frame_metadata_only`. +6. `resubmitting_the_same_batch_reconciles_to_one` — no duplicate rows, no second version, no delete. +7. `a_conflicting_batch_for_a_durable_cycle_fails_closed`. +8. `a_stale_horizon_is_fenced_and_writes_nothing`. +9. `intent_only_casts_leave_no_durable_trace` — mixed cycle persists only its artifact cast. +10. `randomized_completion_order_yields_the_same_durable_set` — identical batch hash + image. +11. `a_malformed_artifact_payload_is_refused` — 511 bytes refused, nothing written. + +`crates/lance-graph-planner/src/persist_sink.rs` adds the contract-level pairs +(`intent_only_cycle_is_nochange_zero_store_calls`, +`retrying_the_same_batch_reconciles_never_duplicates`, +`a_different_batch_for_a_committed_cycle_fails_closed`, +`randomized_completion_order_yields_the_same_batch_hash`, +`after_cycle_bound_limits_the_sealed_scan`). + +### Deferred with reasons (not silently skipped) + +- **Real S3/object-store commit + ambiguous-response paths** — no credentials in + this environment. **The capability IS compiled in**: S3 requires lance's + `aws` feature, and `lance = "=9.0.0"` is taken with default features (which + include `aws`, `azure`, `gcp`), verified in the dependency graph + (`aws-config`/`aws-credential-types` reach `lance-io` → `lance` → + `lance-graph`). `LanceCycleWriter::open` takes any URI `Dataset::open` + accepts, so an `s3://` store compiles and routes today. What is unproven is + the credentialed RUN — commit, reconciliation after a lost response, and the + bounded tail read against a real bucket. **No object-store durability claim + is made** until that falsifier executes; a deployment enabling S3 must not + read "the aws feature is on" as "the path is measured". +- **`Continue`-from-pulse with zero Lance reads** — Phase C (the pulse + machinery does not exist yet). +- **64k-scale gather/commit measurement** — the contract is proven at contract + scale here; the 64k arm belongs with the measurement harness. diff --git a/.claude/plans/persistence-cycle-wal-bootstrap-v1.md b/.claude/plans/persistence-cycle-wal-bootstrap-v1.md index b3dc3aa56..3f255bd4d 100644 --- a/.claude/plans/persistence-cycle-wal-bootstrap-v1.md +++ b/.claude/plans/persistence-cycle-wal-bootstrap-v1.md @@ -1,8 +1,20 @@ # persistence-cycle-wal-bootstrap-v1 — the primitive cycle/WAL seam and its temporal/revision upgrade path +> **⊘ PARTIALLY SUPERSEDED (operator ruling 2026-08-09) — read +> `.claude/plans/persistence-artifact-backed-commit-v1.md` FIRST.** Guarantee 5 +> below ("one successful cycle seal produces exactly one `DatasetVersion`") is +> now conditional: **no artifact-backed semantic change → no write → no new +> version**. An empty / intent-only cycle produces `CommitOutcome::NoChange` +> with ZERO store operations. Guarantees 1–4 and 6 survive unchanged, and +> guarantee 6 is strengthened (the writer additionally holds no owner borrow +> across the commit await). The §2 sparse-delta rule is now IMPLEMENTED — the +> concrete sink is `lance_graph::graph::cycle_sink::LanceCycleWriter`, whose +> coalesced image rows are the durable end-form. Append-only: nothing below is +> deleted; it is read through the newer contract. +> > **Status:** ACTIVE (bootstrap SHIPPED in PR #878; upgrade phases PLANNED; the -> §2 sparse-delta storage rule is RATIFIED architecture, UNIMPLEMENTED in a -> concrete Lance sink). +> §2 sparse-delta storage rule is RATIFIED architecture, IMPLEMENTED in +> `LanceCycleWriter` as of Phase A 2026-08-09). > **Date:** 2026-08-02. > **Scope:** documentation-only architectural ruling. Records the *role* of the > #878 persistence seam and the intended larger two-dimensional temporal diff --git a/crates/lance-graph-planner/examples/blw_fusion.rs b/crates/lance-graph-planner/examples/blw_fusion.rs index e0be83a43..28271d74f 100644 --- a/crates/lance-graph-planner/examples/blw_fusion.rs +++ b/crates/lance-graph-planner/examples/blw_fusion.rs @@ -97,8 +97,8 @@ use lance_graph_contract::soa_view::{IdentityPlane, MailboxSoaView}; use lance_graph_planner::batch_writer::BatchWriter; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - persist_cycle, recover_and_apply, CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, - SweepSlot, WalSink, WriteFailed, + persist_cycle, recover_and_apply, CommitError, CommitOutcome, CycleFrame, CycleId, + DetachedCycleBatch, FrameMeta, LandedSlot, SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::temporal::{ deinterlace, DeinterlaceRow, LanceVersion, NoDeps, QueryReference, @@ -390,7 +390,8 @@ impl RowSpanDescriptor { /// helper this harness has no caller for; keeping a write-only field would be dead code). struct SealedCycle { cycle: CycleId, - version: DatasetVersion, + base_version: DatasetVersion, + batch_hash: u64, landings: Vec, } @@ -399,7 +400,8 @@ struct SealedCycle { /// (one append per cycle, stored order, sealed read horizon), NOT durability. struct MemWal { sealed: Mutex>, - next_version: AtomicU64, + /// The store's physical head version (0 = empty store). + head: AtomicU64, wal_writes: AtomicU64, } @@ -407,7 +409,7 @@ impl MemWal { fn new() -> Self { Self { sealed: Mutex::new(Vec::new()), - next_version: AtomicU64::new(1), + head: AtomicU64::new(0), wal_writes: AtomicU64::new(0), } } @@ -415,63 +417,83 @@ impl MemWal { self.wal_writes.load(Ordering::SeqCst) } fn head(&self) -> DatasetVersion { - self.sealed - .lock() - .expect("MemWal poisoned") - .last() - .map_or(DatasetVersion(0), |s| s.version) + DatasetVersion(self.head.load(Ordering::SeqCst)) } } impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); - let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + // Reconciliation-first: an already-durable (cycle, hash) is success, a + // matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: DatasetVersion(self.head.load(Ordering::SeqCst)), + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } + let head = DatasetVersion(self.head.load(Ordering::SeqCst)); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); - let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let version = DatasetVersion(self.head.fetch_add(1, Ordering::SeqCst) + 1); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedCycle { - cycle: batch.frame.cycle, - version, + cycle, + base_version: batch.frame.base_version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| (s.cycle, s.version)) + .map(|s| FrameMeta { + cycle: s.cycle, + base_version: s.base_version, + batch_hash: s.batch_hash, + }) .collect()) } } @@ -735,7 +757,7 @@ async fn main() -> Result<(), Box> { owner.set_populated(seated_total); owner.tick(); // cycle 0 -> 1, mirrors blw_tenant.rs:618. - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut writer: BatchWriter = BatchWriter::new(); let mut watermark: Option = None; @@ -869,14 +891,27 @@ async fn main() -> Result<(), Box> { }]; let appends_before = sink.wal_writes(); let base = sink.head(); - let version = persist_cycle(&sink, CycleFrame::new(spec.id, base), slots).await?; + let commit_outcome = + persist_cycle(&mut sink, CycleFrame::new(spec.id, base), slots).await?; + let CommitOutcome::Committed { version, .. } = commit_outcome else { + panic!( + "cycle {:?}: expected a fresh Committed outcome (every cycle id in this \ + harness's plan is used exactly once), got {commit_outcome:?}", + spec.id + ); + }; assert_eq!( sink.wal_writes() - appends_before, 1, "one landing -> exactly ONE WAL append" ); - let sealed = sink.scan_sealed(Some(base)).await?; + // `after_cycle` bounds the read to cycles strictly after the PREVIOUS + // cycle in `plan` (cycle ids are the 1-indexed plan position), which is + // exactly this cycle's own newly-sealed landing — the cycle-keyed + // equivalent of the old `scan_sealed(Some(base))` version-keyed bound. + let after_cycle = (spec.id.0 > 1).then(|| CycleId(spec.id.0 - 1)); + let sealed = sink.scan_sealed(after_cycle).await?; let recovered = recover_and_apply(&mut owner, &sealed, watermark).map_err(|(_, e)| e)?; watermark = recovered.watermark; assert_eq!( diff --git a/crates/lance-graph-planner/examples/blw_tenant.rs b/crates/lance-graph-planner/examples/blw_tenant.rs index 075b0291d..16dfd8e4e 100644 --- a/crates/lance-graph-planner/examples/blw_tenant.rs +++ b/crates/lance-graph-planner/examples/blw_tenant.rs @@ -94,8 +94,8 @@ use lance_graph_contract::soa_view::{IdentityPlane, MailboxSoaOwner, MailboxSoaV use lance_graph_planner::batch_writer::BatchWriter; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - persist_cycle, recover_and_apply, CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, - SweepSlot, WalSink, WriteFailed, + persist_cycle, recover_and_apply, CommitError, CommitOutcome, CycleFrame, CycleId, + DetachedCycleBatch, FrameMeta, LandedSlot, SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::traits::StrategyOutcome; @@ -394,8 +394,10 @@ impl RowSpanDescriptor { struct SealedCycle { /// The cycle identity. cycle: CycleId, - /// The version the seal published. - version: DatasetVersion, + /// The sealed predecessor this cycle's thoughts read (`Vn`). + base_version: DatasetVersion, + /// The batch's durable idempotency hash. + batch_hash: u64, /// Landings AS COMMITTED — already write-side ordered before the append. landings: Vec, /// Distinct rows in the coalesced final image. @@ -403,14 +405,16 @@ struct SealedCycle { } /// An in-process `WalSink`, mirroring `persist_sink`'s own `FakeWalSink` -/// (including its optimistic-concurrency fence on `base`). +/// (including its optimistic-concurrency fence on the frame's `base_version` +/// and its reconciliation-first `(cycle, batch_hash)` lookup). /// /// **This proves the CONTRACT (one append per cycle, stored order, sealed read /// horizon), NOT durability.** There is no WAL, no restart, no manifest, and no /// Lance table anywhere in this file. struct MemWal { sealed: Mutex>, - next_version: AtomicU64, + /// The store's physical head version (0 = empty store). + head: AtomicU64, /// Physical appends — must be exactly ONE per committed cycle. wal_writes: AtomicU64, } @@ -419,7 +423,7 @@ impl MemWal { fn new() -> Self { Self { sealed: Mutex::new(Vec::new()), - next_version: AtomicU64::new(1), + head: AtomicU64::new(0), wal_writes: AtomicU64::new(0), } } @@ -427,11 +431,7 @@ impl MemWal { self.wal_writes.load(Ordering::SeqCst) } fn head(&self) -> DatasetVersion { - self.sealed - .lock() - .expect("MemWal poisoned") - .last() - .map_or(DatasetVersion(0), |s| s.version) + DatasetVersion(self.head.load(Ordering::SeqCst)) } fn image_rows_of(&self, cycle: CycleId) -> Option { self.sealed @@ -445,33 +445,53 @@ impl MemWal { impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); + // Reconciliation-first: an already-durable (cycle, hash) is success, a + // matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: DatasetVersion(self.head.load(Ordering::SeqCst)), + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } // Epistemic horizon: a commit must target the current sealed head. - let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + let head = DatasetVersion(self.head.load(Ordering::SeqCst)); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } // THE single amortized append for the whole cycle. self.wal_writes.fetch_add(1, Ordering::SeqCst); - let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let version = DatasetVersion(self.head.fetch_add(1, Ordering::SeqCst) + 1); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedCycle { - cycle: batch.frame.cycle, - version, + cycle, + base_version: batch.frame.base_version, + batch_hash, image_rows: batch.image.len(), landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { // Returned in STORED order — never re-sorted (order was fixed at seal). Ok(self @@ -479,23 +499,27 @@ impl WalSink for MemWal { .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| (s.cycle, s.version)) + .map(|s| FrameMeta { + cycle: s.cycle, + base_version: s.base_version, + batch_hash: s.batch_hash, + }) .collect()) } } @@ -719,7 +743,7 @@ async fn main() -> Result<(), Box> { ); // ── the sealed cycles ──────────────────────────────────────────────────── - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut writer: BatchWriter = BatchWriter::new(); let mut watermark: Option = None; let mut stream_position: u64 = 0; @@ -886,7 +910,14 @@ async fn main() -> Result<(), Box> { let appends_before = sink.wal_writes(); let base = sink.head(); - let version = persist_cycle(&sink, CycleFrame::new(spec.id, base), slots).await?; + let outcome = persist_cycle(&mut sink, CycleFrame::new(spec.id, base), slots).await?; + let CommitOutcome::Committed { version, .. } = outcome else { + panic!( + "cycle {:?}: expected a fresh Committed outcome (every cycle id in this \ + harness's plan is used exactly once), got {outcome:?}", + spec.id + ); + }; assert_eq!( sink.wal_writes() - appends_before, 1, @@ -899,7 +930,12 @@ async fn main() -> Result<(), Box> { .map(|m| m.to); // ── ⑥ the post-seal apply — THE PAIRED MOVE, via try_advance_phase ── - let sealed = sink.scan_sealed(Some(base)).await?; + // `after_cycle` bounds the read to cycles strictly after the PREVIOUS + // cycle in this plan (cycle ids are the 1-indexed plan position), which + // is exactly this cycle's own newly-sealed landings — the cycle-keyed + // equivalent of the old `scan_sealed(Some(base))` version-keyed bound. + let after_cycle = (spec.id.0 > 1).then(|| CycleId(spec.id.0 - 1)); + let sealed = sink.scan_sealed(after_cycle).await?; let pre_apply = snapshot(&owner); let recovered = recover_and_apply(&mut owner, &sealed, watermark).map_err(|(_, e)| e)?; watermark = recovered.watermark; @@ -1029,7 +1065,7 @@ async fn main() -> Result<(), Box> { ); // ── summary ────────────────────────────────────────────────────────────── - let versions = sink.versions().await?; + let frames = sink.timeline().await?; println!("--"); println!( "tenants : 1 (never N) — mailbox {}, final phase {:?}, cycle {}", @@ -1040,7 +1076,7 @@ async fn main() -> Result<(), Box> { println!( "seal : {} cycles → {} versions, {} WAL appends, watermark {:?}", plan.len(), - versions.len(), + frames.len(), sink.wal_writes(), watermark ); diff --git a/crates/lance-graph-planner/src/persist_sink.rs b/crates/lance-graph-planner/src/persist_sink.rs index 4b7805c56..161decc16 100644 --- a/crates/lance-graph-planner/src/persist_sink.rs +++ b/crates/lance-graph-planner/src/persist_sink.rs @@ -61,12 +61,53 @@ //! ## What the tests prove — CONTRACT probes, NOT crash/WAL integration (honest) //! //! The `#[cfg(test)]` `FakeWalSink` is in-process maps, not a WAL/Lance table. The -//! tests are **storage/race CONTRACT probes**: one WAL write per cycle, write-side -//! order under scrambled completion, per-row coalescing, no unsealed visibility, -//! sealed read horizon, recovery idempotence. They do **NOT** prove real -//! durability (no MemWAL, restart, atomic `RecordBatch`, or manifest). -//! **`compile+test green ≠ storage proven`** (the Ladybug lesson). **This module -//! builds NO concrete Lance sink.** +//! tests are **storage/race CONTRACT probes**: one durable commit per cycle, +//! write-side order under scrambled completion, per-row coalescing, no unsealed +//! visibility, sealed read horizon, recovery idempotence. They do **NOT** prove +//! real durability. **`compile+test green ≠ storage proven`** (the Ladybug +//! lesson). The concrete sink is `lance_graph::graph::cycle_sink::LanceCycleWriter`. +//! +//! ## The governing storage rule (operator-ruled 2026-08-09 — supersedes the +//! ## earlier "one version per cycle, empty cycles included" contract) +//! +//! **No artifact-backed semantic change → no write → no new [`DatasetVersion`].** +//! +//! - A cast with a NON-EMPTY payload is an **artifact cast** — a real semantic +//! delta (grounding result, reusable walk product, adjudication, terminal +//! conclusion). Only these persist. +//! - A cast with an EMPTY payload is an **intent-only cast** (a held-intent +//! re-stage, a pure kanban step). It is EPHEMERAL: its move still applies to +//! the in-memory fleet, but it never reaches the store, never trips a +//! payload gate, and after a crash is cheaply regenerated from the pinned +//! sealed inputs. Kanban never decides when to persist; the anyway-happening +//! artifact write is what carries kanban progress into durability. +//! - A cycle with ZERO artifact casts is [`CommitOutcome::NoChange`]: zero +//! store calls, zero rows, unchanged head. (The predecessor contract's +//! deliberate empty-cycle versioning is REMOVED, not repaired.) +//! +//! **Honest limit of the Phase-A gate (review-flagged):** the gate tests +//! payload PRESENCE, not semantic CHANGE — an identical artifact re-emitted in +//! a later cycle still commits (a new cycle genuinely concluded, but with +//! unchanged content). Change-detection needs conclusion IDENTITY (the +//! `(episode, plan, revision)` model + producer-side digest dedup) and is the +//! Phase-D conclusion-boundary refinement; a typed +//! `IntentOnly | ArtifactChanged` cast kind belongs there too, replacing the +//! empty-payload CONVENTION with a type. Until then this doc — not the type +//! system — is what says an empty payload means intent. +//! +//! ## One logical writer — commit is `&mut self`, no rollback, no delete +//! +//! There is exactly ONE logical application writer per cycle store; the 64k +//! thoughts are parallel PRODUCERS, not writers. The trait therefore commits +//! through `&mut self` (two application commits cannot interleave through the +//! type boundary), and a published manifest is HISTORY — there is no +//! compensating delete and no rollback. An unexpected head is a +//! fence/reconciliation condition ([`CommitError::Fenced`] / +//! [`CommitOutcome::Reconciled`]), never normal competition. Idempotency is +//! durable: every committed batch carries its `(cycle, batch_hash)` in the +//! same commit, so a lost acknowledgement reconciles by re-submitting the SAME +//! frozen batch — [`WalSink::commit_cycle`] finds it and returns +//! [`CommitOutcome::Reconciled`] instead of appending twice. use lance_graph_contract::kanban::{KanbanColumn, KanbanMove, RubiconTransitionError}; use lance_graph_contract::scheduler::DatasetVersion; @@ -150,16 +191,130 @@ pub struct SweepSlot { pub payload: Vec, } -/// A landing read back from a SEALED cycle — its slot plus the version its cycle -/// sealed into (shared by every landing of that cycle). Only ever produced for -/// SEALED cycles; returned in the stored (already canonical) order, NEVER re-sorted. +/// A landing read back from a SEALED cycle — its slot plus the cycle it sealed +/// with. Only ever produced for SEALED cycles; returned in the stored (already +/// canonical) order, NEVER re-sorted. +/// +/// Keyed by CYCLE, not by dataset version: the physical [`DatasetVersion`] is +/// the publication position of the sole writer's commit (returned in +/// [`CommitOutcome::Committed`]) and is deliberately NOT re-derivable per row — +/// semantic identity lives in `(cycle, batch_hash)`. #[derive(Debug, Clone, PartialEq, Eq)] pub struct LandedSlot { - /// The version the cycle sealed into (the cheap coarse timeline). - pub version: DatasetVersion, + /// The cycle this landing sealed with (the recovery bound / grouping key). + pub cycle: CycleId, pub slot: SweepSlot, } +/// One committed cycle's frame metadata — the coarse timeline row. Compact: +/// never carries payloads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FrameMeta { + pub cycle: CycleId, + /// The sealed read horizon this cycle's thoughts read (`Vn`). + pub base_version: DatasetVersion, + /// The batch's deterministic content hash (the durable idempotency key). + pub batch_hash: u64, +} + +/// The honest result states of a commit attempt. There is NO state meaning +/// "nothing landed" that can be returned after publication may have occurred — +/// that case either reconciles to [`Reconciled`](CommitOutcome::Reconciled) or +/// surfaces as [`CommitError::Ambiguous`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CommitOutcome { + /// Zero artifact casts: zero store calls, zero rows, unchanged head. + NoChange { head: DatasetVersion }, + /// The batch is durable at the ACTUAL returned physical PUBLICATION + /// version — the audit-grade reference (unlike + /// [`Reconciled`](CommitOutcome::Reconciled)'s `current_head`). + Committed { + version: DatasetVersion, + cycle: CycleId, + batch_hash: u64, + }, + /// The batch was ALREADY durable (a lost acknowledgement / retry): found by + /// its `(cycle, batch_hash)` identity; no second append happened. + /// + /// **`current_head` is deliberately NOT named `version`**: it is the store + /// head AT RECONCILIATION TIME, not the version this cycle originally + /// published at (retrying cycle 1 after cycle 5 reconciles at head V5). + /// An audit trail must reference the durable identity + /// `(cycle, batch_hash)` — the exact publication position is recoverable + /// from the version history on the audit path, never inferred from this + /// field. Only [`Committed`](CommitOutcome::Committed)'s `version` is a + /// publication version. + Reconciled { + current_head: DatasetVersion, + cycle: CycleId, + batch_hash: u64, + }, +} + +/// Why a commit attempt did NOT yield a durable outcome. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CommitError { + /// The store head is not the frame's sealed predecessor and this batch is + /// not already durable. NOTHING was written: regenerate against the + /// current head. Under the one-writer topology this means a lost-response + /// restart, a stale cached head, an unauthorized writer, or corruption — + /// never normal competition. + Fenced { current_head: DatasetVersion }, + /// This cycle is durable with a DIFFERENT batch hash — fail closed + /// (corruption / identity conflict). Never promoted, never overwritten. + HashConflict { + cycle: CycleId, + stored_hash: u64, + offered_hash: u64, + }, + /// I/O failed with provably NOTHING published — safe to regenerate from + /// the unchanged horizon. + Io(WriteFailed), + /// The append's outcome could not be determined AND reconciliation itself + /// failed. Re-submit the SAME frozen batch: `commit_cycle` reconciles + /// first, so the retry cannot double-append. + Ambiguous { + cycle: CycleId, + batch_hash: u64, + cause: String, + }, +} + +impl std::fmt::Display for CommitError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Fenced { current_head } => { + write!(f, "fenced: store head is {current_head:?}, nothing written") + } + Self::HashConflict { + cycle, + stored_hash, + offered_hash, + } => write!( + f, + "cycle {cycle:?} durable with hash {stored_hash:#018x}, offered {offered_hash:#018x} — fail closed" + ), + Self::Io(e) => write!(f, "commit I/O (nothing published): {e}"), + Self::Ambiguous { + cycle, + batch_hash, + cause, + } => write!( + f, + "AMBIGUOUS: cycle {cycle:?} (hash {batch_hash:#018x}) may or may not be durable: {cause}" + ), + } + } +} +impl std::error::Error for CommitError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Io(e) => Some(e), + _ => None, + } + } +} + /// A durable write that did not land (the cycle seal failed / was fenced). #[derive(Debug, Clone, PartialEq, Eq)] pub struct WriteFailed(pub String); @@ -179,19 +334,29 @@ impl std::error::Error for WriteFailed {} #[derive(Debug, Clone, PartialEq, Eq)] pub struct DetachedCycleBatch { pub frame: CycleFrame, - /// Landings in canonical `stream_position` order (the temporal deinterlace - /// already ran — the WAL never sees worker-completion order). + /// ARTIFACT landings (non-empty payloads) in canonical `stream_position` + /// order (the temporal deinterlace already ran — the store never sees + /// worker-completion order). Intent-only casts never enter a batch. pub landings: Vec, /// The coalesced final image: `row -> last payload in stream order`. pub image: std::collections::BTreeMap>, + /// Deterministic content hash over the CANONICAL landings — the durable + /// idempotency key. Identical completed sets yield identical hashes + /// regardless of worker completion order (the freeze canonicalizes first). + pub batch_hash: u64, } impl DetachedCycleBatch { - /// Freeze concurrently-produced casts into the ordered, coalesced cycle image: - /// (1) [`order_cycle_stably`] stable-orders by `stream_position` — - /// completion order never becomes storage order (physical race); (2) fold - /// same-row updates into owned rows (later stream position wins). The result is - /// detached from any live SoA and ready for exactly one WAL append. + /// Freeze concurrently-produced ARTIFACT casts into the ordered, coalesced + /// cycle image: (1) [`order_cycle_stably`] stable-orders by + /// `stream_position` — completion order never becomes storage order + /// (physical race); (2) fold same-row updates into owned rows (later + /// stream position wins); (3) hash the canonical content. The result is + /// detached from any live SoA and ready for exactly one durable commit. + /// + /// Callers pass artifact casts only ([`persist_cycle`] partitions); + /// passing intent-only casts here would persist control flow, which the + /// governing storage rule forbids. #[must_use] pub fn freeze(frame: CycleFrame, mut casts: Vec) -> Self { order_cycle_stably(&mut casts, |s| s.stream_position); @@ -199,12 +364,45 @@ impl DetachedCycleBatch { for s in &casts { image.insert(s.row, s.payload.clone()); } + let batch_hash = Self::content_hash(frame, &casts); Self { frame, landings: casts, image, + batch_hash, } } + + /// FNV-1a 64 over the frame identity + canonical landing content. + fn content_hash(frame: CycleFrame, canonical: &[SweepSlot]) -> u64 { + const OFFSET: u64 = 0xcbf2_9ce4_8422_2325; + const PRIME: u64 = 0x0000_0100_0000_01b3; + let mut h = OFFSET; + let mut eat = |bytes: &[u8]| { + for b in bytes { + h ^= u64::from(*b); + h = h.wrapping_mul(PRIME); + } + }; + eat(&frame.cycle.0.to_le_bytes()); + eat(&frame.base_version.0.to_le_bytes()); + for s in canonical { + eat(&s.stream_position.to_le_bytes()); + eat(&s.owner.to_le_bytes()); + eat(&s.row.to_le_bytes()); + match &s.paired_move { + Some(m) => eat(&[1, m.from as u8, m.to as u8, m.exec as u8]), + None => eat(&[0]), + } + if let Some(m) = &s.paired_move { + eat(&m.mailbox.to_le_bytes()); + eat(&m.witness_chain_position.to_le_bytes()); + } + eat(&(s.payload.len() as u64).to_le_bytes()); + eat(&s.payload); + } + h + } } /// Why a persist / seal / step operation produced no lifecycle advance. @@ -213,6 +411,13 @@ pub enum PersistError { /// The cycle seal did not land — nothing published, so no step. This is the /// RETRYABLE class (a fenced / failed append may succeed on retry). Write(WriteFailed), + /// The durable commit did not yield an outcome — see [`CommitError`] for + /// the honest sub-states ([`Fenced`](CommitError::Fenced) = regenerate + /// against the new head; [`Io`](CommitError::Io) = nothing landed, safe + /// regenerate; [`Ambiguous`](CommitError::Ambiguous) = re-submit the SAME + /// frozen batch, reconciliation decides; [`HashConflict`](CommitError::HashConflict) + /// = fail closed). + Commit(CommitError), /// A cast was staged against a different cycle than the frame — a caller /// programming error, PERMANENT and never retryable (distinct from [`Write`] /// so retry logic never loops on it). [`Write`]: PersistError::Write @@ -242,6 +447,7 @@ impl std::fmt::Display for PersistError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { Self::Write(e) => write!(f, "{e}"), + Self::Commit(e) => write!(f, "{e}"), Self::CycleMismatch { cast_cycle, frame_cycle, @@ -277,6 +483,7 @@ impl std::error::Error for PersistError { fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { match self { Self::Write(e) => Some(e), + Self::Commit(e) => Some(e), Self::Illegal(e) => Some(e), Self::CycleMismatch { .. } | Self::OwnerMismatch { .. } | Self::StalePhase { .. } => { None @@ -301,45 +508,67 @@ impl std::error::Error for PersistError { /// to be spawned, give it explicit `impl Future + Send` methods there. #[allow(async_fn_in_trait)] pub trait WalSink { - /// The SINGLE amortized WAL append for a whole cycle: commit `batch` (already - /// ordered + coalesced) against the sealed predecessor `base`, publishing - /// exactly one new [`DatasetVersion`] atomically. This is the ONLY durable op — - /// 64k thoughts → one append, not 64k appends. `async` because the concrete WAL - /// append is async. + /// The SINGLE amortized durable commit for a whole cycle: commit `batch` + /// (already ordered + coalesced + hashed, artifact casts only) against the + /// sealed predecessor `batch.frame.base_version`, publishing exactly one + /// new [`DatasetVersion`] atomically — 64k thoughts → one commit. + /// + /// **`&mut self` — the one-logical-writer boundary.** Two application + /// commits cannot interleave through this type boundary (a concrete writer + /// is additionally non-`Clone`). Producers stay fire-and-forget on their + /// own `&self`/staging surfaces; only the sole writer drives this method + /// and it FULLY honors the result. + /// + /// Reconciliation-first: an implementation looks the batch's + /// `(cycle, batch_hash)` up BEFORE appending, so re-submitting the same + /// frozen batch after a lost acknowledgement returns + /// [`CommitOutcome::Reconciled`] instead of double-appending. There is no + /// rollback and no compensating delete: a published manifest is history. async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result; + ) -> Result; /// Read back COMMITTED landings (never an uncommitted cycle), in the STORED - /// canonical order — this seam does NOT sort. Optionally only cycles after - /// `from_version`. + /// canonical order — this seam does NOT sort. `after_cycle` bounds the read + /// to cycles strictly after it (the recovery tail bound); implementations + /// push the bound into the storage scan, never full-scan-and-filter. async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed>; - /// The CHEAP coarse timeline: `(CycleId, DatasetVersion)` per committed cycle, - /// WITHOUT replaying any landing. The read a downstream time-series consumer - /// (e.g. `stockfish-rs`, another session) does — a lookup of the version table - /// over already-coherent frames. - async fn versions(&self) -> Result, WriteFailed>; + /// The CHEAP coarse timeline: one [`FrameMeta`] per committed cycle, + /// WITHOUT touching any payload. NOT a normal-path read — after a + /// successful commit the caller already holds the outcome; this is for + /// recovery, audit, and downstream consumers. + async fn timeline(&self) -> Result, WriteFailed>; } -/// Deinterlace + freeze a whole cycle's casts and commit it in ONE WAL append. -/// The loom ([`order_cycle_stably`]) and the freeze run in -/// [`DetachedCycleBatch::freeze`] BEFORE the single [`WalSink::commit_cycle`], so -/// completion order never reaches storage. Rejects a cross-owner move (and a -/// cycle-id mismatch) before the batch is built. Returns the one published version. +/// Partition, deinterlace + freeze a cycle's ARTIFACT casts and commit them in +/// ONE durable operation — or perform NONE when there is nothing artifact-backed. +/// +/// The governing storage rule, applied here: +/// - Intent-only casts (EMPTY payload — held-intent re-stages, pure kanban +/// steps) are partitioned OUT before the freeze. They stay ephemeral: their +/// moves still reach the fleet through the caller's transition set, but no +/// payload gate sees them and no store row carries them. +/// - Zero artifact casts ⇒ [`CommitOutcome::NoChange`], with the sink NEVER +/// called: zero store operations, zero rows, unchanged head. +/// - Otherwise the loom ([`order_cycle_stably`]) and freeze run in +/// [`DetachedCycleBatch::freeze`] BEFORE the single +/// [`WalSink::commit_cycle`], so completion order never reaches storage. +/// +/// Rejects a cross-owner move (and a cycle-id mismatch) across ALL casts — +/// ephemeral ones included — before anything is built. pub async fn persist_cycle( - sink: &S, + sink: &mut S, frame: CycleFrame, casts: Vec, -) -> Result { +) -> Result { for c in &casts { if c.cycle != frame.cycle { - // Permanent caller error — NOT a retryable Write (retry would loop). + // Permanent caller error — NOT a retryable class (retry would loop). return Err(PersistError::CycleMismatch { cast_cycle: c.cycle, frame_cycle: frame.cycle, @@ -354,11 +583,19 @@ pub async fn persist_cycle( } } } - // Loom-before-WAL: order + coalesce + detach, THEN one atomic append. - let batch = DetachedCycleBatch::freeze(frame, casts); - sink.commit_cycle(frame.base_version, batch) - .await - .map_err(PersistError::Write) + // The artifact gate: only non-empty payloads are semantic deltas. + let artifacts: Vec = casts + .into_iter() + .filter(|c| !c.payload.is_empty()) + .collect(); + if artifacts.is_empty() { + return Ok(CommitOutcome::NoChange { + head: frame.base_version, + }); + } + // Loom-before-commit: order + coalesce + hash + detach, THEN one commit. + let batch = DetachedCycleBatch::freeze(frame, artifacts); + sink.commit_cycle(batch).await.map_err(PersistError::Commit) } /// The result of a [`recover_and_apply`] pass: the moves applied this pass, and the @@ -508,20 +745,21 @@ mod tests { } } - // ── The WAL sink fake ──────────────────────────────────────────────────────── - struct SealedCycle { + // ── The store fake (contract probe; concrete = LanceCycleWriter) ──────────── + struct SealedRec { frame: CycleFrame, - version: DatasetVersion, - /// The landings AS COMMITTED (already write-side ordered before the append). + batch_hash: u64, + /// The landings AS COMMITTED (already write-side ordered before the commit). landings: Vec, /// The coalesced final image: row -> last value in stream order. image: BTreeMap>, } struct FakeWalSink { succeed: bool, - sealed: Mutex>, - next_version: AtomicU64, - /// The number of physical WAL appends — must be ONE per committed cycle. + sealed: Mutex>, + /// The store's physical head version (0 = empty store). + head: AtomicU64, + /// The number of physical durable commits — must be ONE per committed cycle. wal_writes: AtomicU64, } impl FakeWalSink { @@ -529,7 +767,7 @@ mod tests { Self { succeed: true, sealed: Mutex::new(Vec::new()), - next_version: AtomicU64::new(1), + head: AtomicU64::new(0), wal_writes: AtomicU64::new(0), } } @@ -542,6 +780,9 @@ mod tests { fn wal_writes(&self) -> u64 { self.wal_writes.load(Ordering::SeqCst) } + fn head(&self) -> DatasetVersion { + DatasetVersion(self.head.load(Ordering::SeqCst)) + } fn version_count(&self) -> usize { self.sealed.lock().unwrap().len() } @@ -555,12 +796,12 @@ mod tests { } /// Test-only: inject a committed cycle whose landings are DELIBERATELY out /// of order, to prove `scan_sealed` does not re-sort (order is a write-side - /// property, fixed before the append — never a read-time repair). + /// property, fixed before the commit — never a read-time repair). fn inject_unordered_committed(&self, frame: CycleFrame, landings: Vec) { - let v = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); - self.sealed.lock().unwrap().push(SealedCycle { + self.head.fetch_add(1, Ordering::SeqCst); + self.sealed.lock().unwrap().push(SealedRec { frame, - version: v, + batch_hash: 0, landings, image: BTreeMap::new(), }); @@ -568,38 +809,55 @@ mod tests { } impl WalSink for FakeWalSink { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { if !self.succeed { - return Err(WriteFailed("cycle fenced".into())); + return Err(CommitError::Io(WriteFailed("cycle store I/O".into()))); } let mut sealed = self.sealed.lock().unwrap(); - // Optimistic-concurrency fence: a commit MUST target the current sealed - // head (`Vn`). A stale/in-flight `base` (a sibling that read an older - // predecessor) is rejected — the epistemic horizon enforced, not assumed. - let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: DatasetVersion(self.head.load(Ordering::SeqCst)), + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } + // The fence: a commit MUST target the current head (`Vn`). Under the + // one-writer topology a mismatch is a stale horizon, never competition. + let head = DatasetVersion(self.head.load(Ordering::SeqCst)); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } - // The batch arrives ALREADY deinterlaced + coalesced (loom before WAL). - // THE single amortized WAL append for the whole cycle. + // THE single amortized durable commit for the whole cycle. self.wal_writes.fetch_add(1, Ordering::SeqCst); - let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); - sealed.push(SealedCycle { + let version = DatasetVersion(self.head.fetch_add(1, Ordering::SeqCst) + 1); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); + sealed.push(SealedRec { frame: batch.frame, - version, + batch_hash, landings: batch.landings, image: batch.image, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { // Returned in STORED order — no sort. (The order was fixed at seal.) Ok(self @@ -607,22 +865,26 @@ mod tests { .lock() .unwrap() .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .unwrap() .iter() - .map(|s| (s.frame.cycle, s.version)) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, + }) .collect()) } } @@ -657,31 +919,41 @@ mod tests { // ── FALSIFIER (headline): one WAL write per cycle — the amortization ───────── #[tokio::test] async fn a_whole_cycle_of_casts_is_one_wal_write_one_version() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let frame = CycleFrame::new(CycleId(1), DatasetVersion(0)); // 100 concurrent thoughts stage into an owned cast vector — building it - // touches NO WAL (staging is the caller's, not the sink's). + // touches NO store (staging is the caller's, not the sink's). let casts: Vec = (0..100u64).map(|i| slot(42, 1, i, i, None)).collect(); assert_eq!( sink.wal_writes(), 0, - "staging writes no WAL — pure amortization" + "staging writes nothing — pure amortization" ); assert_eq!(sink.version_count(), 0, "no version before the seal"); - // persist_cycle is the SINGLE amortized WAL write for the whole cycle. - let version = persist_cycle(&sink, frame, casts).await.unwrap(); - assert_eq!(sink.wal_writes(), 1, "100 casts → exactly ONE WAL write"); - assert_eq!(version, DatasetVersion(1), "→ exactly one version"); + // persist_cycle is the SINGLE amortized durable commit for the cycle. + let out = persist_cycle(&mut sink, frame, casts).await.unwrap(); + assert_eq!(sink.wal_writes(), 1, "100 casts → exactly ONE commit"); + assert!( + matches!( + out, + CommitOutcome::Committed { + version: DatasetVersion(1), + cycle: CycleId(1), + .. + } + ), + "→ exactly one version, honestly reported: {out:?}" + ); assert_eq!(sink.version_count(), 1); } // ── FALSIFIER: write-side order under scrambled completion (physical race) ─── #[tokio::test] async fn scrambled_completion_lands_in_canonical_stream_order_at_write_time() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); // Thoughts "finish" (stage) in scrambled CPU order: stream 2, 0, 3, 1. - let sealed = persist_cycle( - &sink, + persist_cycle( + &mut sink, CycleFrame::new(CycleId(5), DatasetVersion(0)), vec![ slot(42, 5, 2, 20, None), @@ -698,7 +970,7 @@ mod tests { .await .unwrap() .iter() - .filter(|l| l.version == sealed) + .filter(|l| l.cycle == CycleId(5)) .map(|l| l.slot.stream_position) .collect(); assert_eq!( @@ -739,9 +1011,9 @@ mod tests { // ── FALSIFIER: per-row coalescing (not last-chunk-wins) ────────────────────── #[tokio::test] async fn same_row_updates_coalesce_distinct_rows_survive() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(3), DatasetVersion(0)), vec![ // Two updates to ROW 7 (stream 0 then 2) — coalesce to the later. @@ -775,7 +1047,7 @@ mod tests { // ── FALSIFIER: no partial visibility — an unsealed cycle is invisible ──────── #[tokio::test] async fn an_unsealed_cycle_is_invisible_epistemic_horizon() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); // A cycle's casts are staged in an owned vector held by the caller — NOT in // the sink. Until the single atomic commit_cycle runs, nothing is visible: // an open cycle's output is NOT readable as Vn input (read Vn / write Vn+1). @@ -792,13 +1064,17 @@ mod tests { "before the seal, no landing is visible", ); assert!( - sink.versions().await.unwrap().is_empty(), - "no version before seal" + sink.timeline().await.unwrap().is_empty(), + "no frame before seal" ); // The seal is all-or-nothing: the whole cycle appears at once, never partially. - persist_cycle(&sink, CycleFrame::new(CycleId(9), DatasetVersion(0)), casts) - .await - .unwrap(); + persist_cycle( + &mut sink, + CycleFrame::new(CycleId(9), DatasetVersion(0)), + casts, + ) + .await + .unwrap(); assert_eq!( sink.scan_sealed(None).await.unwrap().len(), 1, @@ -809,28 +1085,37 @@ mod tests { // ── FALSIFIER: sealed read horizon — a cycle reads the sealed predecessor ──── #[tokio::test] async fn a_cycle_reads_the_sealed_predecessor_not_an_in_flight_sibling() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); // Cycle 1 seals → V1 (the sealed head advances to V1). let s1 = persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![slot(42, 1, 0, 0, None)], ) .await .unwrap(); - assert_eq!(s1, DatasetVersion(1)); + let CommitOutcome::Committed { version: v1, .. } = s1 else { + panic!("expected Committed, got {s1:?}"); + }; + assert_eq!(v1, DatasetVersion(1)); // A sibling that read the STALE predecessor V0 (an in-flight base, not the - // sealed head V1) is FENCED — the sink rejects the commit, proving the - // horizon is enforced, not merely restated by the caller. + // sealed head V1) is FENCED — the sink refuses the commit with the + // current head, writing NOTHING (never normal competition under the + // one-writer topology; a fence/reconciliation condition). let stale = persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(2), DatasetVersion(0)), vec![slot(99, 2, 0, 0, None)], ) .await; assert!( - matches!(stale, Err(PersistError::Write(_))), - "committing against a stale/in-flight base is fenced, not silently accepted", + matches!( + stale, + Err(PersistError::Commit(CommitError::Fenced { + current_head: DatasetVersion(1) + })) + ), + "committing against a stale base is Fenced with the current head: {stale:?}", ); assert_eq!( sink.version_count(), @@ -839,45 +1124,62 @@ mod tests { ); // Committing against the SEALED head V1 succeeds and publishes exactly V2. let s2 = persist_cycle( - &sink, - CycleFrame::new(CycleId(2), s1), + &mut sink, + CycleFrame::new(CycleId(2), v1), vec![slot(99, 2, 0, 0, None)], ) .await .unwrap(); - assert_eq!(s2, DatasetVersion(2), "and publishes exactly its own Vn+1"); + assert!( + matches!( + s2, + CommitOutcome::Committed { + version: DatasetVersion(2), + .. + } + ), + "and publishes exactly its own Vn+1: {s2:?}" + ); } // ── FALSIFIER: cheap time-series — version table only, no landing replay ───── #[tokio::test] async fn downstream_time_series_reads_the_version_table_not_the_landings() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); for c in 1..=3u64 { persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(c), DatasetVersion(c - 1)), vec![slot(42, c, 0, 0, None)], ) .await .unwrap(); } + let frames = sink.timeline().await.unwrap(); assert_eq!( - sink.versions().await.unwrap(), + frames + .iter() + .map(|f| (f.cycle, f.base_version)) + .collect::>(), vec![ - (CycleId(1), DatasetVersion(1)), - (CycleId(2), DatasetVersion(2)), - (CycleId(3), DatasetVersion(3)), + (CycleId(1), DatasetVersion(0)), + (CycleId(2), DatasetVersion(1)), + (CycleId(3), DatasetVersion(2)), ], - "history scales with CYCLES; a time-series consumer just looks up the version table", + "history scales with CYCLES; the timeline is frame metadata, no payloads", + ); + assert!( + frames.iter().all(|f| f.batch_hash != 0), + "every frame carries its durable idempotency hash" ); } // ── Recovery over sealed landings (post-seal only) ─────────────────────────── #[tokio::test] async fn recovery_replays_an_owners_chain_in_stream_order_skipping_others() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(4), DatasetVersion(0)), vec![ slot( @@ -919,9 +1221,9 @@ mod tests { #[tokio::test] async fn cyclic_recovery_is_idempotent_only_with_the_watermark() { assert_eq!(KanbanColumn::Plan.next_phases(), &[KanbanColumn::Planning]); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![ slot( @@ -987,9 +1289,9 @@ mod tests { // A mixed chain: step@0, a NO-STEP landing@1, step@2. If the no-step // landing did not advance the watermark, it would be re-scanned forever and // block the watermark behind every following step. - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![ slot( @@ -999,7 +1301,7 @@ mod tests { 0, Some((KanbanColumn::Planning, KanbanColumn::CognitiveWork)), ), - slot(42, 1, 1, 1, None), // no-step landing at position 1 + slot(42, 1, 1, 1, None), // no-step (artifact) landing at position 1 slot( 42, 1, @@ -1031,9 +1333,9 @@ mod tests { // step@0 is valid and advances the owner; step@1 has a `from` that no longer // matches (corruption) → StalePhase. The Err must carry the partial Recovered // so the caller can persist the watermark for the prefix that DID apply. - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![ slot( @@ -1085,9 +1387,9 @@ mod tests { // stream_position is monotonic per owner ACROSS cycles: cycle 1 carries // position 0, cycle 2 carries position 1. Recovery over the multi-cycle // scan_sealed stream must treat the watermark as cross-cycle. - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![slot( 42, @@ -1100,7 +1402,7 @@ mod tests { .await .unwrap(); persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(2), DatasetVersion(1)), vec![slot( 42, @@ -1145,9 +1447,9 @@ mod tests { async fn a_cast_for_the_wrong_cycle_is_a_permanent_cycle_mismatch_not_write() { // A cast whose cycle != frame.cycle is a caller programming error — it must // NOT surface as the RETRYABLE Write class (retry would loop forever). - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let r = persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![slot(42, 2, 0, 0, None)], // cast cycle 2 ≠ frame cycle 1 ) @@ -1175,10 +1477,10 @@ mod tests { paired_move: Some(mv(99, KanbanColumn::Planning, KanbanColumn::CognitiveWork)), ..slot(42, 1, 0, 0, None) }; - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); assert!(matches!( persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![bad] ) @@ -1194,9 +1496,9 @@ mod tests { #[tokio::test] async fn a_fenced_cycle_writes_nothing() { - let sink = FakeWalSink::failing(); + let mut sink = FakeWalSink::failing(); let r = persist_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), vec![slot( 42, @@ -1207,8 +1509,8 @@ mod tests { )], ) .await; - assert!(matches!(r, Err(PersistError::Write(_)))); - assert_eq!(sink.wal_writes(), 0, "no WAL write, no version, no step"); + assert!(matches!(r, Err(PersistError::Commit(CommitError::Io(_))))); + assert_eq!(sink.wal_writes(), 0, "no commit, no version, no step"); assert_eq!(sink.scan_sealed(None).await.unwrap().len(), 0); } @@ -1228,4 +1530,182 @@ mod tests { order_cycle_stably(&mut rows, |r| r.0); assert_eq!(rows, vec![(1, "b1"), (1, "b2"), (2, "c"), (3, "d")]); } + + // ── FALSIFIER (Phase A): no artifact-backed delta → zero store operations ──── + #[tokio::test] + async fn intent_only_cycle_is_nochange_zero_store_calls() { + let mut sink = FakeWalSink::new(); + // A cycle of PURE kanban movement: every cast is intent-only (empty + // payload — the restage_held shape). Kanban never decides to persist. + let intents: Vec = (0..5u64) + .map(|i| SweepSlot { + payload: Vec::new(), + ..slot( + 42, + 1, + i, + i, + Some((KanbanColumn::Planning, KanbanColumn::CognitiveWork)), + ) + }) + .collect(); + let out = persist_cycle( + &mut sink, + CycleFrame::new(CycleId(1), DatasetVersion(7)), + intents, + ) + .await + .unwrap(); + assert_eq!( + out, + CommitOutcome::NoChange { + head: DatasetVersion(7) + }, + "pure movement is NoChange with the unchanged head" + ); + assert_eq!(sink.wal_writes(), 0, "zero store operations"); + assert_eq!(sink.version_count(), 0, "zero rows, zero frames"); + // And a MIXED cycle persists ONLY its artifact casts. + let mixed = vec![ + SweepSlot { + payload: Vec::new(), + ..slot( + 42, + 2, + 10, + 1, + Some((KanbanColumn::Planning, KanbanColumn::CognitiveWork)), + ) + }, + slot(42, 2, 11, 2, None), // artifact (non-empty payload) + ]; + let out = persist_cycle( + &mut sink, + CycleFrame::new(CycleId(2), DatasetVersion(0)), + mixed, + ) + .await + .unwrap(); + assert!(matches!(out, CommitOutcome::Committed { .. })); + let sealed = sink.scan_sealed(None).await.unwrap(); + assert_eq!(sealed.len(), 1, "only the artifact cast persisted"); + assert_eq!(sealed[0].slot.stream_position, 11); + } + + // ── FALSIFIER (Phase A): retrying the same frozen batch reconciles ─────────── + #[tokio::test] + async fn retrying_the_same_batch_reconciles_never_duplicates() { + let mut sink = FakeWalSink::new(); + let casts = || vec![slot(42, 1, 0, 0, None), slot(42, 1, 1, 1, None)]; + let first = persist_cycle( + &mut sink, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + casts(), + ) + .await + .unwrap(); + assert!(matches!(first, CommitOutcome::Committed { .. })); + // The acknowledgement was "lost"; the caller re-submits the SAME batch. + let retry = persist_cycle( + &mut sink, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + casts(), + ) + .await + .unwrap(); + assert!( + matches!( + retry, + CommitOutcome::Reconciled { + cycle: CycleId(1), + .. + } + ), + "the retry reconciles instead of appending twice: {retry:?}" + ); + assert_eq!(sink.wal_writes(), 1, "exactly one physical commit"); + assert_eq!( + sink.scan_sealed(None).await.unwrap().len(), + 2, + "no duplicate landings" + ); + } + + // ── FALSIFIER (Phase A): a conflicting batch for a committed cycle fails closed + #[tokio::test] + async fn a_different_batch_for_a_committed_cycle_fails_closed() { + let mut sink = FakeWalSink::new(); + persist_cycle( + &mut sink, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![slot(42, 1, 0, 0, None)], + ) + .await + .unwrap(); + // Same cycle identity, DIFFERENT content — corruption/conflict, never promoted. + let conflict = persist_cycle( + &mut sink, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![slot(42, 1, 5, 5, None)], + ) + .await; + assert!( + matches!( + conflict, + Err(PersistError::Commit(CommitError::HashConflict { + cycle: CycleId(1), + .. + })) + ), + "{conflict:?}" + ); + assert_eq!(sink.wal_writes(), 1, "the conflicting batch wrote nothing"); + } + + // ── FALSIFIER (Phase A): completion order never reaches the batch identity ─── + #[test] + fn randomized_completion_order_yields_the_same_batch_hash() { + let frame = CycleFrame::new(CycleId(3), DatasetVersion(2)); + let ordered = vec![ + slot(1, 3, 0, 0, None), + slot(2, 3, 1, 1, None), + slot(3, 3, 2, 2, None), + ]; + let scrambled = vec![ + slot(3, 3, 2, 2, None), + slot(1, 3, 0, 0, None), + slot(2, 3, 1, 1, None), + ]; + let a = DetachedCycleBatch::freeze(frame, ordered); + let b = DetachedCycleBatch::freeze(frame, scrambled); + assert_eq!( + a.batch_hash, b.batch_hash, + "identity comes from canonical content" + ); + assert_eq!(a.landings, b.landings, "identical canonical landings"); + // And DIFFERENT content yields a different hash (the hash discriminates). + let c = DetachedCycleBatch::freeze(frame, vec![slot(1, 3, 0, 9, None)]); + assert_ne!(a.batch_hash, c.batch_hash); + } + + // ── FALSIFIER (Phase A): the recovery tail bound excludes earlier cycles ───── + #[tokio::test] + async fn after_cycle_bound_limits_the_sealed_scan() { + let mut sink = FakeWalSink::new(); + for c in 1..=3u64 { + persist_cycle( + &mut sink, + CycleFrame::new(CycleId(c), DatasetVersion(c - 1)), + vec![slot(42, c, c, c, None)], + ) + .await + .unwrap(); + } + let tail = sink.scan_sealed(Some(CycleId(1))).await.unwrap(); + assert_eq!( + tail.iter().map(|l| l.cycle).collect::>(), + vec![CycleId(2), CycleId(3)], + "strictly after the bound — bounded recovery, not full history" + ); + } } diff --git a/crates/lance-graph-supervisor/examples/measure_wal_curve.rs b/crates/lance-graph-supervisor/examples/measure_wal_curve.rs index 305254ecd..0e7de05df 100644 --- a/crates/lance-graph-supervisor/examples/measure_wal_curve.rs +++ b/crates/lance-graph-supervisor/examples/measure_wal_curve.rs @@ -92,8 +92,8 @@ mod measure { use lance_graph_planner::ir::Arena; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - order_cycle_stably, persist_cycle, CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, - SweepSlot, WalSink, WriteFailed, + order_cycle_stably, persist_cycle, CommitError, CommitOutcome, CycleFrame, CycleId, + DetachedCycleBatch, FrameMeta, LandedSlot, SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::strategy::style_strategy::StyleStrategy; use lance_graph_planner::temporal::{ @@ -789,9 +789,16 @@ mod measure { .map(|s| s.stream_position + 1) .max() .unwrap_or(0); - let _ = frame; // frame carried by the caller's own bookkeeping only + // Synthetic outcome — this helper never calls `WalSink::commit_cycle` + // (see the doc above), so there is no real `batch_hash` to carry; `0` + // is a placeholder, never compared against a real committed hash. DriverSealedCycle { - version, + outcome: CommitOutcome::Committed { + version, + cycle: frame.cycle, + batch_hash: 0, + }, + version: Some(version), transitions, next_position_base, } @@ -1407,7 +1414,11 @@ mod measure { // ═════════════════════════════════════════════════════════════════════ struct SealedEntry { + frame: CycleFrame, version: DatasetVersion, + /// The batch's deterministic content hash — the reconciliation-first + /// idempotency key `commit_cycle` looks up BEFORE appending. + batch_hash: u64, landings: Vec, } @@ -1439,56 +1450,76 @@ mod measure { impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: rec.version, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedEntry { + frame: batch.frame, version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| { - ( - s.landings.first().map_or(CycleId(0), |l| l.cycle), - s.version, - ) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, }) .collect()) } @@ -1537,7 +1568,7 @@ mod measure { // (`SweepSlot::paired_move = None` — a sanctioned landing shape per // `persist_sink.rs:145-147`'s own doc). Real `persist_cycle` calls // against a real (in-process) `WalSink`, not a fabricated Vec. - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut position_base: u64 = 0; for cyc in 1..=16u64 { let mut writer: BatchWriter> = BatchWriter::new(); @@ -1548,7 +1579,7 @@ mod measure { assert_eq!(collected.slots.len(), FLEET_OWNERS as usize); let base = sink.head(); let frame = CycleFrame::new(CycleId(cyc), base); - persist_cycle(&sink, frame, collected.slots) + persist_cycle(&mut sink, frame, collected.slots) .await .unwrap_or_else(|e| panic!("temporal history: cycle {cyc} failed to seal: {e}")); position_base += u64::from(FLEET_OWNERS); @@ -1580,7 +1611,13 @@ mod measure { .map(|ls| BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + // `LandedSlot` is keyed by CYCLE, not physical `DatasetVersion` + // (persist_sink's governing storage rule: a cycle with only + // intent-only casts publishes no version at all). This + // benchmark's cycles are 1:1 with commits (every cast carries + // a non-empty payload, so every cycle here IS a version), so + // `cycle.0` is the same monotonic identity `version.0` was. + lance_version: ls.cycle.0, }) .collect(); @@ -2183,14 +2220,18 @@ mod measure { "EXP-KIA: one move per owner, nothing held" ); - let sink = MemWal::new(); + let mut sink = MemWal::new(); let frame = CycleFrame::new(CycleId(1), DatasetVersion(0)); let t_wal = Instant::now(); - let version = persist_cycle(&sink, frame, collected.slots.clone()) + let outcome = persist_cycle(&mut sink, frame, collected.slots.clone()) .await .expect("EXP-KIA: seal must succeed"); let wal_write_ns = t_wal.elapsed().as_nanos() as u64; assert_eq!(sink.wal_writes(), 1, "EXP-KIA: exactly one WAL commit"); + let version = match outcome { + CommitOutcome::Committed { version, .. } => version, + other => panic!("EXP-KIA: every cast has a non-empty payload, expected Committed, got {other:?}"), + }; assert_eq!(version, DatasetVersion(1)); let sealed = build_sealed_locally(frame, &collected.slots, version); @@ -2450,7 +2491,10 @@ mod measure { out.entry(ls.slot.owner).or_default().push(BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + // See run_temporal's identical substitution note: `cycle.0` + // stands in for the retired `version.0` (1:1 here, every + // cast carries a non-empty payload). + lance_version: ls.cycle.0, }); } Ok(out) @@ -2484,7 +2528,7 @@ mod measure { wal_path: &std::path::Path, ) -> (MArmPhaseMedians, Vec, MemWal) { let style_outcome = build_style_outcome(); - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut file = OpenOptions::new() .create(true) .write(true) @@ -2596,7 +2640,7 @@ mod measure { // slots is safe because its internal `order_cycle_stably` is a // no-op-preserving STABLE sort of already-sorted input. let slots_for_commit = frozen.landings.clone(); - persist_cycle(&sink, frame, slots_for_commit) + persist_cycle(&mut sink, frame, slots_for_commit) .await .unwrap_or_else(|e| panic!("M-arm: cycle {cyc} failed to seal: {e}")); @@ -2649,9 +2693,10 @@ mod measure { // write/seal timing to settle), but T1 must cover exactly the // MEASURED window or its number is not comparable to A0's // 78-86 ms over 1,048,576 rows — and beating that number is the - // whole point of the ordered fast path. `scan_sealed(Some(v))` - // filters `version > v`, and the warm-ups own versions 1..=WARMUP. - let after_warmup = Some(DatasetVersion(WARMUP_CYCLES as u64)); + // whole point of the ordered fast path. `scan_sealed(Some(c))` + // filters `cycle > c` (bounded recovery is the contract now, not + // `DatasetVersion`-keyed), and the warm-ups own cycles 1..=WARMUP. + let after_warmup = Some(CycleId(WARMUP_CYCLES as u64)); let landed_natural = natural_sink .scan_sealed(after_warmup) .await @@ -2676,7 +2721,7 @@ mod measure { .map(|ls| BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + lance_version: ls.cycle.0, }) .collect(); let bench_morton: Vec = landed_morton @@ -2684,7 +2729,7 @@ mod measure { .map(|ls| BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + lance_version: ls.cycle.0, }) .collect(); @@ -2735,7 +2780,7 @@ mod measure { // order 2-row input must be REFUSED, not silently accepted. let bad = vec![ LandedSlot { - version: DatasetVersion(2), + cycle: CycleId(2), slot: SweepSlot { cycle: CycleId(2), stream_position: 10, @@ -2746,7 +2791,7 @@ mod measure { }, }, LandedSlot { - version: DatasetVersion(2), + cycle: CycleId(2), slot: SweepSlot { cycle: CycleId(2), stream_position: 5, // regressed — must be refused @@ -2986,7 +3031,7 @@ mod measure { cast_order: &[MailboxId], ) -> (OArmPhaseMedians, MemWal) { let style_outcome = build_style_outcome(); - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut cast_samples = Vec::with_capacity(MEASURED_CYCLES as usize); let mut collect_samples = Vec::with_capacity(MEASURED_CYCLES as usize); let mut order_derive_samples = Vec::with_capacity(MEASURED_CYCLES as usize); @@ -3043,7 +3088,7 @@ mod measure { assert_eq!(frozen.landings.len(), FLEET_OWNERS as usize); let t_commit = Instant::now(); - sink.commit_cycle(frame.base_version, frozen) + sink.commit_cycle(frozen) .await .unwrap_or_else(|e| panic!("O-arm {label}: cycle {cyc} failed to seal: {e}")); let commit_ns = t_commit.elapsed().as_nanos() as u64; @@ -3064,7 +3109,7 @@ mod measure { // cycles' rows. The replay must cover exactly the measured window or // its cost is not comparable to A0's or the M-arm's. let landed = sink - .scan_sealed(Some(DatasetVersion(WARMUP_CYCLES as u64))) + .scan_sealed(Some(CycleId(WARMUP_CYCLES as u64))) .await .expect("O-arm: scan_sealed over the measured window"); assert_eq!( @@ -3077,7 +3122,7 @@ mod measure { .map(|ls| BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + lance_version: ls.cycle.0, }) .collect(); let trajectories = local_trajectories(&bench); @@ -3110,7 +3155,7 @@ mod measure { .map(|ls| BenchRow { owner: ls.slot.owner, cast_seq: ls.slot.stream_position, - lance_version: ls.version.0, + lance_version: ls.cycle.0, }) .collect(); let trajectories = local_trajectories(&bench); diff --git a/crates/lance-graph-supervisor/src/cycle_driver.rs b/crates/lance-graph-supervisor/src/cycle_driver.rs index 3e0fe5b43..a0dbef9da 100644 --- a/crates/lance-graph-supervisor/src/cycle_driver.rs +++ b/crates/lance-graph-supervisor/src/cycle_driver.rs @@ -101,8 +101,8 @@ use lance_graph_contract::QualiaI4_16D; use lance_graph_planner::batch_writer::BatchWriter; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - persist_cycle, recover_and_apply, CycleFrame, CycleId, LandedSlot, PersistError, SweepSlot, - WalSink, + persist_cycle, recover_and_apply, CommitOutcome, CycleFrame, CycleId, LandedSlot, PersistError, + SweepSlot, WalSink, }; use lance_graph_planner::traits::StrategyOutcome; @@ -127,10 +127,23 @@ pub struct SealedTransition { /// is typically ≪ the fleet size. #[derive(Debug, Clone, PartialEq, Eq)] pub struct SealedCycle { - /// The one version this cycle sealed into. - pub version: DatasetVersion, + /// The raw commit result — the honest source of truth. Under the + /// governing storage rule a cycle with zero artifact casts is + /// [`CommitOutcome::NoChange`] and publishes nothing. + pub outcome: CommitOutcome, + /// The version this cycle sealed into, or `None` for + /// [`CommitOutcome::NoChange`] (nothing published, head unchanged). + /// Derived from [`Self::outcome`] at construction — kept as a plain field + /// (not a method) so existing `sealed.version` call sites stay readable; + /// `Some` for [`CommitOutcome::Committed`] / [`CommitOutcome::Reconciled`]. + pub version: Option, /// Only the owners that cast a `paired_move` — the sparse subset. /// With the pre-seal ≤1-per-owner partition, at most one per owner. + /// **Computed over ALL collected casts, not just artifact ones** — an + /// intent-only (empty-payload) cast's move still seals here even when the + /// cycle as a whole is [`CommitOutcome::NoChange`]: kanban movement is + /// ephemeral in the STORE, never in the in-memory fleet (see + /// [`restage_held`] and the governing storage rule in `persist_sink`). pub transitions: Vec, /// The first unused stream position AFTER this cycle, computed over **all** /// sealed slots (including no-move landings). The caller's next @@ -151,6 +164,17 @@ pub struct SealedCycle { /// `casts` via [`seal_cycle`] to skip the re-stage pass; callers that simply /// drop this value are equally correct. This must never grow into a /// provisional-planning ledger. +/// +/// **`cause`'s honest sub-states** (routed from `persist_sink::CommitError` +/// via `PersistError::Commit`): [`CommitError::Fenced`](lance_graph_planner::persist_sink::CommitError::Fenced) +/// and [`CommitError::Io`](lance_graph_planner::persist_sink::CommitError::Io) both mean nothing was published — +/// regenerating from the unchanged `Vn` is always safe. +/// [`CommitError::Ambiguous`](lance_graph_planner::persist_sink::CommitError::Ambiguous) means the outcome is UNKNOWN — do NOT +/// regenerate a fresh cycle; re-submit the SAME frozen `casts` (via +/// [`seal_cycle`]) and let `commit_cycle`'s reconciliation-first lookup +/// decide. [`CommitError::HashConflict`](lance_graph_planner::persist_sink::CommitError::HashConflict) is a fail-closed corruption/identity +/// conflict — never promoted, never overwritten; this driver mints no new +/// recovery machinery for it, it simply surfaces the error. #[derive(Debug)] pub struct SealFailure { /// The frame the failed seal was submitted under (regenerate with the same @@ -166,8 +190,11 @@ pub struct SealFailure { /// P4b output — the effect of applying a sealed cycle's sparse transition set. #[derive(Debug, Clone, PartialEq, Eq)] pub struct AppliedCycle { - /// The version whose sealed transitions were applied. - pub version: DatasetVersion, + /// The version whose sealed transitions were applied — mirrors + /// [`SealedCycle::version`]: `None` for a [`CommitOutcome::NoChange`] + /// cycle (an all-intent-only cycle can still apply sparse transitions to + /// the in-memory fleet even though nothing was published to the store). + pub version: Option, /// One move per **advanced** owner (distinct owners; ≤1 per cycle). pub applied: Vec, /// Defence-in-depth counter: same-owner extras in a sealed input NOT @@ -300,6 +327,12 @@ pub fn collect_casts( /// Re-stage held intents ([`CollectedCasts::held`]) into the writer for the /// NEXT cycle. The re-cast is intent-only (empty payload — the original cast's /// payload already sealed with its cycle). Returns the number re-staged. +/// +/// **This empty payload is precisely what makes the re-staged intent +/// EPHEMERAL under the governing storage rule** (`persist_sink`'s artifact +/// gate): an intent-only cast never trips the payload gate and never reaches +/// the store, so no held-intent re-stage can ever accidentally become a +/// persisted artifact. pub fn restage_held(writer: &mut BatchWriter>, held: Vec) -> usize { let n = held.len(); for h in held { @@ -322,12 +355,22 @@ pub fn restage_held(writer: &mut BatchWriter>, held: Vec) -> /// re-staging is equally correct — the regeneration falsifier proves the same /// semantic cycle re-derives). The one clone held until commit success is the /// price of offering that cache. +/// +/// **`sink: &mut S`** — `persist_cycle`'s `WalSink::commit_cycle` is the +/// one-logical-writer boundary (`&mut self`); this function's own +/// `PersistError::Commit` sub-states stay the honest routing they always +/// were: `CommitError::Fenced` / `CommitError::Io` mean nothing published +/// (regenerate from `Vn`); `CommitError::Ambiguous` means UNKNOWN (re-submit +/// the SAME frozen `casts` — reconciliation decides); `CommitError::HashConflict` +/// is fail closed (never promoted, never overwritten). pub async fn seal_cycle( - sink: &S, + sink: &mut S, frame: CycleFrame, casts: Vec, ) -> Result> { // Read the transitions + next base out BEFORE `persist_cycle` takes ownership. + // Computed over ALL collected casts (artifact AND intent-only) — the + // in-memory fleet step is not gated by the store's artifact filter. let mut transitions: Vec = casts .iter() .filter_map(|s| { @@ -347,11 +390,21 @@ pub async fn seal_cycle( // Held until commit success — the price of a byte-identical retry. let frozen = casts.clone(); match persist_cycle(sink, frame, casts).await { - Ok(version) => Ok(SealedCycle { - version, - transitions, - next_position_base, - }), + Ok(outcome) => { + let version = match outcome { + CommitOutcome::NoChange { .. } => None, + CommitOutcome::Committed { version, .. } => Some(version), + // `current_head` is the store head AT RECONCILIATION TIME, not + // the publication version this cycle originally committed at. + CommitOutcome::Reconciled { current_head, .. } => Some(current_head), + }; + Ok(SealedCycle { + outcome, + version, + transitions, + next_position_base, + }) + } Err(cause) => Err(Box::new(SealFailure { frame, casts: frozen, @@ -484,15 +537,34 @@ pub enum CycleError { } /// Convenience: run one full cycle — **P4a** (drain the writer + seal) then -/// **P4b** (apply the sparse set + advance watermarks). The one seam a running -/// loop calls per cycle. +/// **P4b** (apply the sparse set + advance watermarks) — as ONE sequential +/// call. +/// +/// **Borrow honesty (operator-flagged):** this signature holds the exclusive +/// `&mut F` fleet borrow for the ENTIRE future — including across the seal's +/// storage `.await` — because a future captures its parameters for its whole +/// lifetime regardless of when they are dereferenced. A caller awaiting +/// `run_cycle` therefore cannot touch the fleet concurrently with storage +/// I/O. This makes `run_cycle` the SEQUENTIAL convenience (tests, probes, +/// single-task loops). The PRODUCTION detached path is the split the pieces +/// already expose: `collect_casts` (writer only) → drop the fleet borrow → +/// `seal_cycle(&mut sink, …).await` (sink only, NO fleet parameter) → +/// re-acquire the fleet → `apply_sealed_transitions` (fleet only, no sink). +/// Unrelated thoughts keep computing during the await because nothing they +/// need is borrowed. /// /// `position_base` is the durable stream cursor (see [`collect_casts`]); /// `watermarks` is the fleet's per-owner recovery watermark map, advanced in /// place alongside the phases. On [`CycleError::Seal`] the frozen cycle is /// retryable; on [`CycleError::Apply`] the applied prefix is preserved. +/// +/// **Borrow note (operator-ruled): `fleet: &mut F` is not touched across the +/// seal's `.await`.** The parameter's lifetime spans the whole function, but +/// the body only reads/writes through it in [`apply_sealed_transitions`], +/// AFTER `seal_cycle`'s I/O has already completed — the exclusive fleet +/// borrow is effectively taken post-I/O, not held live across the WAL commit. pub async fn run_cycle( - sink: &S, + sink: &mut S, fleet: &mut F, writer: &mut BatchWriter>, frame: CycleFrame, @@ -505,6 +577,8 @@ where F: MailboxFleet, { let collected = collect_casts(writer, frame.cycle, position_base, row_of); + // `fleet` is untouched up to and including this await — the seal's I/O + // runs before the fleet is ever resolved (see the borrow note above). let sealed = seal_cycle(sink, frame, collected.slots) .await .map_err(CycleError::Seal)?; @@ -730,6 +804,16 @@ pub struct FleetRecovery { pub total_applied: usize, /// Owners that had a pending tail replayed (non-empty applied set). pub owners_recovered: usize, + /// Landings observed in the scanned tail for owners NOT in this pass's + /// `fleet_ids` (or absent from the fleet) — latecomers whose recovery is + /// still owed. `0` means the scanned tail belonged entirely to this pass. + pub foreign_landings: usize, + /// The SMALLEST cycle carrying such a foreign landing. **The latecomer + /// fence:** the caller must NEVER raise its durable `after_cycle` bound to + /// or past this cycle until those owners have recovered — advancing the + /// global bound over an unrecovered latecomer's tail silences it + /// permanently. `None` = no foreign landings in the scanned tail. + pub foreign_min_cycle: Option, } /// **P4e — COMMITTED-HISTORY recovery ONLY.** Valid solely when a commit @@ -748,6 +832,14 @@ pub struct FleetRecovery { /// are skipped; unrepresented owners are untouched. `watermarks` is updated in /// place with the new per-owner watermark to persist alongside the SoA phase. /// +/// **Bounded recovery is now the contract.** `after_cycle` is threaded +/// straight into [`WalSink::scan_sealed`]'s own bound: the scan reads only +/// cycles strictly after it, a BOUNDED tail — never the unbounded full sealed +/// history. Callers durably track the last cycle they fully recovered through +/// (e.g. the highest [`SealedCycle`] cycle whose transitions were applied) and +/// pass it back in on the next recovery pass; pass `None` only for a +/// from-scratch recovery over the whole sealed history. +/// /// On a mid-owner failure the partial progress is kept: the failing owner's /// watermark is still advanced for its applied prefix (per /// `recover_and_apply`'s `Err((partial, cause))` contract) before the error is @@ -757,12 +849,16 @@ pub async fn recover_fleet( fleet: &mut F, fleet_ids: &[MailboxId], watermarks: &mut HashMap>, + after_cycle: Option, ) -> Result where S: WalSink, F: MailboxFleet, { - let sealed: Vec = sink.scan_sealed(None).await.map_err(PersistError::Write)?; + let sealed: Vec = sink + .scan_sealed(after_cycle) + .await + .map_err(PersistError::Write)?; // Partition once: per-owner tails in stored order (O(history), then each // owner replays only its own tail). let mut by_owner: HashMap> = HashMap::new(); @@ -771,6 +867,21 @@ where } let mut total_applied = 0usize; let mut owners_recovered = 0usize; + // Latecomer accounting: landings owned by mailboxes this pass does NOT + // recover (not listed, or listed but absent from the fleet). Their + // smallest cycle is the floor below which the caller's global + // `after_cycle` bound must stay until they recover. + let ids: std::collections::HashSet = fleet_ids.iter().copied().collect(); + let mut foreign_landings = 0usize; + let mut foreign_min_cycle: Option = None; + for (owner_id, slots) in &by_owner { + if !ids.contains(owner_id) || fleet.owner(*owner_id).is_none() { + foreign_landings += slots.len(); + if let Some(mc) = slots.iter().map(|l| l.cycle).min() { + foreign_min_cycle = Some(foreign_min_cycle.map_or(mc, |cur: CycleId| cur.min(mc))); + } + } + } for &id in fleet_ids { let Some(owner) = fleet.owner_mut(id) else { continue; @@ -797,6 +908,8 @@ where Ok(FleetRecovery { total_applied, owners_recovered, + foreign_landings, + foreign_min_cycle, }) } @@ -805,7 +918,9 @@ mod tests { use super::*; use lance_graph_contract::kanban::{ExecTarget, KanbanColumn}; use lance_graph_contract::soa_view::MailboxSoaView; - use lance_graph_planner::persist_sink::{DetachedCycleBatch, LandedSlot, WriteFailed}; + use lance_graph_planner::persist_sink::{ + CommitError, CommitOutcome, DetachedCycleBatch, FrameMeta, LandedSlot, WriteFailed, + }; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::Mutex; @@ -873,14 +988,17 @@ mod tests { struct SealedRec { frame: CycleFrame, version: DatasetVersion, + /// The batch's deterministic content hash — the reconciliation-first + /// idempotency key `commit_cycle` looks up BEFORE appending. + batch_hash: u64, landings: Vec, } struct FakeWalSink { sealed: Mutex>, next_version: AtomicU64, wal_writes: AtomicU64, - reads: AtomicU64, // scan_sealed + versions — MUST stay 0 across P4a+P4b - fail_next: AtomicBool, // injects ONE retryable WAL failure + reads: AtomicU64, // scan_sealed + timeline — MUST stay 0 across P4a+P4b + fail_next: AtomicBool, // injects ONE retryable (Io) WAL failure } impl FakeWalSink { fn new() -> Self { @@ -904,30 +1022,54 @@ mod tests { } impl WalSink for FakeWalSink { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { if self.fail_next.swap(false, Ordering::SeqCst) { - return Err(WriteFailed("injected retryable WAL failure".into())); + return Err(CommitError::Io(WriteFailed( + "injected retryable WAL failure".into(), + ))); } let mut sealed = self.sealed.lock().unwrap(); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: rec.version, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!("stale base {base:?}, head {head:?}"))); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedRec { frame: batch.frame, version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from: Option, + after_cycle: Option, ) -> Result, WriteFailed> { self.reads.fetch_add(1, Ordering::SeqCst); Ok(self @@ -935,23 +1077,27 @@ mod tests { .lock() .unwrap() .iter() - .filter(|s| from.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { self.reads.fetch_add(1, Ordering::SeqCst); Ok(self .sealed .lock() .unwrap() .iter() - .map(|s| (s.frame.cycle, s.version)) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, + }) .collect()) } } @@ -982,7 +1128,7 @@ mod tests { // ── P4a FALSIFIER: drain N casts → exactly one WAL write, one version ─────── #[tokio::test] async fn p4a_drains_casts_and_seals_one_wal_write_one_version() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let owners: Vec = (0..100).collect(); let mut w = writer_with_moves(&owners); let collected = collect_casts(&mut w, CycleId(1), 0, u64::from); @@ -991,14 +1137,18 @@ mod tests { assert_eq!(sink.wal_writes(), 0, "collecting writes no WAL"); let sealed = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) .await .unwrap(); assert_eq!(sink.wal_writes(), 1, "100 casts → exactly ONE WAL write"); - assert_eq!(sealed.version, DatasetVersion(1), "→ exactly one version"); + assert_eq!( + sealed.version, + Some(DatasetVersion(1)), + "→ exactly one version" + ); assert_eq!( sealed.transitions.len(), 100, @@ -1022,7 +1172,7 @@ mod tests { // first batch is deliberately NOT asserted. #[tokio::test] async fn pre_commit_failure_discards_everything_and_regenerates_from_vn() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([ (3, FakeOwner::at(3, KanbanColumn::Planning)), (8, FakeOwner::at(8, KanbanColumn::Planning)), @@ -1054,7 +1204,7 @@ mod tests { let mut w1 = stage(&fleet); sink.fail_next_commit(); let err = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w1, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1085,7 +1235,7 @@ mod tests { // staging — nothing retained from the failed attempt). let mut w2 = stage(&fleet); let out = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w2, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1110,7 +1260,7 @@ mod tests { // Exactly one Vn+1; exactly the represented owners advance once. assert_eq!(sink.wal_writes(), 1, "exactly one successful WAL write"); - assert_eq!(out.sealed.version, DatasetVersion(1)); + assert_eq!(out.sealed.version, Some(DatasetVersion(1))); assert_eq!(out.applied.applied.len(), 2); assert_eq!(fleet[&3].phase(), KanbanColumn::CognitiveWork); assert_eq!(fleet[&8].phase(), KanbanColumn::CognitiveWork); @@ -1121,7 +1271,7 @@ mod tests { // cache gets a byte-identical resubmit. Convenience path, not the contract. #[tokio::test] async fn failed_seal_preserves_the_frozen_cycle_for_byte_identical_retry() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([(9, FakeOwner::at(9, KanbanColumn::Planning))]); let before = fleet.clone(); @@ -1130,7 +1280,7 @@ mod tests { sink.fail_next_commit(); let err = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1150,10 +1300,10 @@ mod tests { let frozen_copy = failure.casts.clone(); // Retry submits the SAME frozen cycle → exactly one version lands. - let sealed = seal_cycle(&sink, failure.frame, failure.casts) + let sealed = seal_cycle(&mut sink, failure.frame, failure.casts) .await .expect("retry succeeds"); - assert_eq!(sealed.version, DatasetVersion(1)); + assert_eq!(sealed.version, Some(DatasetVersion(1))); assert_eq!(sink.wal_writes(), 1, "one successful WAL write total"); // Byte-identical: what landed is exactly the frozen set. let landed = sink.scan_sealed(None).await.unwrap(); @@ -1168,7 +1318,7 @@ mod tests { // ── RESTART FALSIFIER: stream positions stay monotonic across writer rebuilds ─ #[tokio::test] async fn restart_stable_stream_positions_survive_writer_reconstruction() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([(5, FakeOwner::at(5, KanbanColumn::Planning))]); let mut wm: HashMap> = HashMap::new(); @@ -1177,7 +1327,7 @@ mod tests { let mut w1 = writer_with_moves(&[5]); let c1 = collect_casts(&mut w1, CycleId(1), 0, u64::from); let s1 = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), c1.slots, ) @@ -1202,7 +1352,7 @@ mod tests { "restart-stable: NOT the raw CastId 0" ); seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(2), DatasetVersion(1)), c2.slots, ) @@ -1212,7 +1362,7 @@ mod tests { // Crash AFTER cycle 2 sealed but BEFORE it applied: recovery from the // cycle-1 watermark must NOT skip the cycle-2 landing. (With the raw // CastId scheme its position would be 0 ≤ watermark 0 → silently lost.) - let rec = recover_fleet(&sink, &mut fleet, &[5], &mut wm) + let rec = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, None) .await .unwrap(); assert_eq!(rec.total_applied, 1, "the later landing was replayed"); @@ -1223,7 +1373,7 @@ mod tests { // ── WATERMARK-COUPLING FALSIFIER: normal apply advances recovery watermarks ─ #[tokio::test] async fn normal_apply_advances_the_recovery_watermark_no_replay_after_crash() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([(5, FakeOwner::at(5, KanbanColumn::Planning))]); let mut wm: HashMap> = HashMap::new(); @@ -1231,7 +1381,7 @@ mod tests { // Normal path: seal + apply. Watermark advances WITH the phase. run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1246,7 +1396,7 @@ mod tests { // Crash + recovery with the SAME persisted watermarks: nothing replays, // no StalePhase — the normal path and recovery share one rule. - let rec = recover_fleet(&sink, &mut fleet, &[5], &mut wm) + let rec = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, None) .await .expect("recovery after a normal apply must not StalePhase-stall"); assert_eq!(rec.total_applied, 0, "already-applied move NOT replayed"); @@ -1267,11 +1417,11 @@ mod tests { let before = fleet.clone(); let mut wm: HashMap> = HashMap::new(); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut w = writer_with_moves(&represented); let collected = collect_casts(&mut w, CycleId(1), 0, u64::from); let sealed = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) @@ -1289,7 +1439,7 @@ mod tests { assert_eq!(applied.applied.len(), 17, "exactly 17 owners advanced"); assert_eq!(applied.deferred, 0); assert_eq!(applied.missing, 0); - assert_eq!(applied.version, DatasetVersion(1)); + assert_eq!(applied.version, Some(DatasetVersion(1))); assert_eq!(wm.len(), 17, "exactly 17 watermarks advanced"); // Every represented owner is now at CognitiveWork (cycle bumped); every @@ -1335,14 +1485,14 @@ mod tests { let before = fleet.clone(); let mut wm: HashMap> = HashMap::new(); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut w: BatchWriter> = BatchWriter::new(); for id in 0..1_000u32 { w.cast(id, vec![], vec![0x00]); // no move → no transition } let collected = collect_casts(&mut w, CycleId(1), 0, u64::from); let sealed = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) @@ -1365,7 +1515,7 @@ mod tests { let mut fleet: HashMap = HashMap::from([(42, FakeOwner::at(42, KanbanColumn::Planning))]); let mut wm: HashMap> = HashMap::new(); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); // Two casts for owner 42 staged in one cycle. let mut w: BatchWriter> = BatchWriter::new(); w.cast( @@ -1392,7 +1542,7 @@ mod tests { ); let sealed = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) @@ -1411,7 +1561,7 @@ mod tests { // Recovery from scratch applies EXACTLY the same set as normal op did. let mut fresh = HashMap::from([(42, FakeOwner::at(42, KanbanColumn::Planning))]); let mut wm2: HashMap> = HashMap::new(); - let rec = recover_fleet(&sink, &mut fresh, &[42], &mut wm2) + let rec = recover_fleet(&mut sink, &mut fresh, &[42], &mut wm2, None) .await .unwrap(); assert_eq!(rec.total_applied, 1, "recovery applies the same ONE move"); @@ -1421,7 +1571,7 @@ mod tests { assert_eq!(restage_held(&mut w, collected.held), 1); let c2 = collect_casts(&mut w, CycleId(2), sealed.next_position_base, u64::from); let s2 = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(2), DatasetVersion(1)), c2.slots, ) @@ -1467,7 +1617,12 @@ mod tests { HashMap::from([(7, FakeOwner::at(7, KanbanColumn::CognitiveWork))]); let mut wm: HashMap> = HashMap::new(); let sealed = SealedCycle { - version: DatasetVersion(1), + outcome: CommitOutcome::Committed { + version: DatasetVersion(1), + cycle: CycleId(1), + batch_hash: 0, + }, + version: Some(DatasetVersion(1)), transitions: vec![SealedTransition { stream_position: 0, owner: 7, @@ -1491,7 +1646,12 @@ mod tests { ]); let mut wm: HashMap> = HashMap::new(); let sealed = SealedCycle { - version: DatasetVersion(1), + outcome: CommitOutcome::Committed { + version: DatasetVersion(1), + cycle: CycleId(1), + batch_hash: 0, + }, + version: Some(DatasetVersion(1)), transitions: vec![ SealedTransition { stream_position: 0, @@ -1525,7 +1685,12 @@ mod tests { let mut fleet: HashMap = HashMap::new(); // empty fleet let mut wm: HashMap> = HashMap::new(); let sealed = SealedCycle { - version: DatasetVersion(1), + outcome: CommitOutcome::Committed { + version: DatasetVersion(1), + cycle: CycleId(1), + batch_hash: 0, + }, + version: Some(DatasetVersion(1)), transitions: vec![SealedTransition { stream_position: 0, owner: 99, @@ -1547,11 +1712,11 @@ mod tests { .map(|id| (id, FakeOwner::at(id, KanbanColumn::Planning))) .collect(); let mut wm: HashMap> = HashMap::new(); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut w = writer_with_moves(&[3, 7]); // only owners 3 and 7 produce a move let out = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1562,7 +1727,7 @@ mod tests { .await .unwrap(); - assert_eq!(out.sealed.version, DatasetVersion(1)); + assert_eq!(out.sealed.version, Some(DatasetVersion(1))); assert_eq!( out.applied.applied.len(), 2, @@ -1594,7 +1759,7 @@ mod tests { // ── P4c FALSIFIER: CognitiveWork thought → next-cycle cast → round-trip ───── #[tokio::test] async fn p4c_cognitive_work_casts_the_next_intent_and_round_trips() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([(5, FakeOwner::at(5, KanbanColumn::Planning))]); let mut wm: HashMap> = HashMap::new(); @@ -1602,7 +1767,7 @@ mod tests { // Cycle 1: owner 5 casts Planning→CognitiveWork; the driver applies it. let out1 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1632,7 +1797,7 @@ mod tests { // Cycle 2: the driver drains that cast → seals V2 → applies → owner 5 → Evaluation. let out2 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(2), DatasetVersion(1)), @@ -1642,7 +1807,7 @@ mod tests { ) .await .unwrap(); - assert_eq!(out2.sealed.version, DatasetVersion(2)); + assert_eq!(out2.sealed.version, Some(DatasetVersion(2))); assert_eq!( out2.applied.applied.len(), 1, @@ -1656,7 +1821,7 @@ mod tests { async fn p4d_an_unfinished_owner_never_blocks_a_completed_owners_cast() { // BOTH owners are represented and BOTH enter CognitiveWork; A's thought // stays unfinished (declines), B completes — B's cast lands regardless. - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut fleet: HashMap = HashMap::from([ (1, FakeOwner::at(1, KanbanColumn::Planning)), (2, FakeOwner::at(2, KanbanColumn::Planning)), @@ -1665,7 +1830,7 @@ mod tests { let mut w = writer_with_moves(&[1, 2]); let out = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1698,7 +1863,7 @@ mod tests { // Cycle 2: B advances; A stays in CognitiveWork (unblocked, unfinished). let out2 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(2), DatasetVersion(1)), @@ -1717,15 +1882,64 @@ mod tests { ); } + // ── P4e FALSIFIER: the `after_cycle` bound BITES (inertness both ways) ─────── + /// A knob that changes nothing is decoration. This drives the SAME durable + /// history twice — once unbounded, once bounded past the landing's cycle — + /// and asserts the bound both EXCLUDES (nothing replays, the owner stays + /// put) and, at a lower bound, ADMITS (the move replays). Without both + /// halves, `after_cycle` could be ignored inside `recover_fleet` and every + /// existing test would still pass, since they all pass `None`. + #[tokio::test] + async fn recover_fleet_after_cycle_bound_excludes_and_admits() { + // Seal owner 5's Planning→CognitiveWork move in CYCLE 2. + let mut sink = FakeWalSink::new(); + let mut w = writer_with_moves(&[5]); + let collected = collect_casts(&mut w, CycleId(2), 0, u64::from); + seal_cycle( + &mut sink, + CycleFrame::new(CycleId(2), DatasetVersion(0)), + collected.slots, + ) + .await + .unwrap(); + + // Bounded PAST the landing's cycle → its tail is outside the read. + let mut fleet: HashMap = + HashMap::from([(5, FakeOwner::at(5, KanbanColumn::Planning))]); + let mut wm: HashMap> = HashMap::new(); + let excluded = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, Some(CycleId(2))) + .await + .unwrap(); + assert_eq!( + excluded.total_applied, 0, + "a bound past the landing's cycle excludes it — the bound is not inert" + ); + assert_eq!( + fleet[&5].phase(), + KanbanColumn::Planning, + "and the owner is untouched" + ); + + // Bounded BELOW it (strictly-after semantics) → the same tail replays. + let admitted = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, Some(CycleId(1))) + .await + .unwrap(); + assert_eq!( + admitted.total_applied, 1, + "a lower bound admits the same landing — the bound discriminates" + ); + assert_eq!(fleet[&5].phase(), KanbanColumn::CognitiveWork); + } + // ── P4e FALSIFIER: recovery replays the pending tail, idempotent w/ watermark ─ #[tokio::test] async fn p4e_recover_fleet_replays_pending_tail_idempotent_with_watermark() { // Seal a cycle with owner 5's Planning→CognitiveWork move (a durable landing). - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let mut w = writer_with_moves(&[5]); let collected = collect_casts(&mut w, CycleId(1), 0, u64::from); seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) @@ -1737,7 +1951,7 @@ mod tests { HashMap::from([(5, FakeOwner::at(5, KanbanColumn::Planning))]); let mut wm: HashMap> = HashMap::new(); - let rec = recover_fleet(&sink, &mut fleet, &[5], &mut wm) + let rec = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, None) .await .unwrap(); assert_eq!(rec.total_applied, 1, "the pending move was replayed"); @@ -1745,7 +1959,7 @@ mod tests { assert_eq!(fleet[&5].phase(), KanbanColumn::CognitiveWork); // Re-drive with the returned watermark → idempotent (nothing re-applied). - let again = recover_fleet(&sink, &mut fleet, &[5], &mut wm) + let again = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm, None) .await .unwrap(); assert_eq!( @@ -1756,7 +1970,7 @@ mod tests { // Negative control: watermark LOST → re-driving the already-advanced owner // stalls (from=Planning ≠ phase=CognitiveWork) → the watermark is load-bearing. let mut wm_lost: HashMap> = HashMap::new(); - let stalled = recover_fleet(&sink, &mut fleet, &[5], &mut wm_lost).await; + let stalled = recover_fleet(&mut sink, &mut fleet, &[5], &mut wm_lost, None).await; assert!( matches!(stalled, Err(PersistError::StalePhase { .. })), "without the watermark an acyclic re-drive stalls — watermark is load-bearing" @@ -1790,12 +2004,12 @@ mod tests { let mut fleet = CountingFleet { inner, resolves: 0 }; let mut wm: HashMap> = HashMap::new(); - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); let represented: Vec = (0..DIRTY).map(|i| i * 100).collect(); let mut w = writer_with_moves(&represented); let collected = collect_casts(&mut w, CycleId(1), 0, u64::from); let sealed = seal_cycle( - &sink, + &mut sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), collected.slots, ) @@ -1888,7 +2102,7 @@ mod tests { // ── P4c GATED FALSIFIER: Flow casts + round-trips; Hold is RESCHEDULED ────── #[tokio::test] async fn gated_flow_casts_hold_is_rescheduled_and_wakes_on_a_later_cycle() { - let sink = FakeWalSink::new(); + let mut sink = FakeWalSink::new(); // Owner 5 will FLOW (advances); owner 6 will HOLD (rests, re-polled later). let mut fleet: HashMap = HashMap::from([ (5, FakeOwner::at(5, KanbanColumn::Planning)), @@ -1899,7 +2113,7 @@ mod tests { // Cycle 1: both cast Planning→CognitiveWork; the driver applies both. let out1 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1926,7 +2140,7 @@ mod tests { // Cycle 2: owner 5 advances to Evaluation; owner 6 rested this round. let out2 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(2), DatasetVersion(1)), @@ -1948,7 +2162,7 @@ mod tests { assert!(woken.held_owners.is_empty()); let out3 = run_cycle( - &sink, + &mut sink, &mut fleet, &mut w, CycleFrame::new(CycleId(3), DatasetVersion(2)), diff --git a/crates/lance-graph-supervisor/tests/d_ign_b_lenses.rs b/crates/lance-graph-supervisor/tests/d_ign_b_lenses.rs index b297a844a..319d13cd0 100644 --- a/crates/lance-graph-supervisor/tests/d_ign_b_lenses.rs +++ b/crates/lance-graph-supervisor/tests/d_ign_b_lenses.rs @@ -126,7 +126,8 @@ mod d_ign_b_lenses { use lance_graph_planner::nars::{BeliefArena, CStmt}; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, SweepSlot, WalSink, WriteFailed, + CommitError, CommitOutcome, CycleFrame, CycleId, DetachedCycleBatch, FrameMeta, LandedSlot, + SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::strategy::style_strategy::StyleStrategy; use lance_graph_planner::traits::{ @@ -429,7 +430,11 @@ mod d_ign_b_lenses { // `probe_ignition.rs`'s `MemWal`. ─────────────────────────────────────── struct SealedCycle { + frame: CycleFrame, version: DatasetVersion, + /// The batch's deterministic content hash — the reconciliation-first + /// idempotency key `commit_cycle` looks up BEFORE appending. + batch_hash: u64, landings: Vec, } @@ -458,56 +463,76 @@ mod d_ign_b_lenses { impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: rec.version, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedCycle { + frame: batch.frame, version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| { - ( - s.landings.first().map_or(CycleId(0), |l| l.cycle), - s.version, - ) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, }) .collect()) } @@ -981,7 +1006,7 @@ mod d_ign_b_lenses { // ── main cycle loop: cast/scan/seal/apply, unchanged mechanics, // with the lens embedded in `run_cognitive_work_gated_over`'s // closure (design §1's chosen seam). ──────────────────────────── - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut writer: BatchWriter> = BatchWriter::new(); let mut position_base: u64 = 0; let mut watermarks: HashMap> = HashMap::new(); @@ -1059,7 +1084,7 @@ mod d_ign_b_lenses { let base_version = sink.head(); let outcome: CycleOutcome = match run_cycle( - &sink, + &mut sink, &mut fleet, &mut writer, CycleFrame::new(CycleId(u64::from(c)), base_version), diff --git a/crates/lance-graph-supervisor/tests/probe_ignition.rs b/crates/lance-graph-supervisor/tests/probe_ignition.rs index 5f06b98c3..06f3f7cf3 100644 --- a/crates/lance-graph-supervisor/tests/probe_ignition.rs +++ b/crates/lance-graph-supervisor/tests/probe_ignition.rs @@ -91,7 +91,8 @@ mod probe_ignition { use lance_graph_planner::ir::Arena; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, SweepSlot, WalSink, WriteFailed, + CommitError, CommitOutcome, CycleFrame, CycleId, DetachedCycleBatch, FrameMeta, LandedSlot, + SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::strategy::style_strategy::StyleStrategy; use lance_graph_planner::traits::{ @@ -325,7 +326,11 @@ mod probe_ignition { // `cycle_driver.rs`'s own `#[cfg(test)]` `FakeWalSink`, cited at G3b). ─ struct SealedCycle { + frame: CycleFrame, version: DatasetVersion, + /// The batch's deterministic content hash — the reconciliation-first + /// idempotency key `commit_cycle` looks up BEFORE appending. + batch_hash: u64, landings: Vec, } @@ -333,7 +338,7 @@ mod probe_ignition { sealed: Mutex>, next_version: AtomicU64, wal_writes: AtomicU64, - /// `scan_sealed` + `versions` call count — MUST stay 0 across the + /// `scan_sealed` + `timeline` call count — MUST stay 0 across the /// main loop (P4b reads no dataset; G3b). reads: AtomicU64, } @@ -364,29 +369,50 @@ mod probe_ignition { impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: rec.version, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedCycle { + frame: batch.frame, version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { self.reads.fetch_add(1, Ordering::SeqCst); Ok(self @@ -394,28 +420,27 @@ mod probe_ignition { .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { self.reads.fetch_add(1, Ordering::SeqCst); Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| { - ( - s.landings.first().map_or(CycleId(0), |l| l.cycle), - s.version, - ) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, }) .collect()) } @@ -732,7 +757,7 @@ mod probe_ignition { eprintln!("probe.ignition.G2c: reliability(Analytical)={r_a} != reliability(Creative)={r_c}; same-style reliability is bit-identical"); } - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut writer: BatchWriter> = BatchWriter::new(); let mut position_base: u64 = 0; let mut watermarks: HashMap> = HashMap::new(); @@ -871,7 +896,7 @@ mod probe_ignition { let wal_writes_before = sink.wal_writes(); let base_version = sink.head(); let outcome: CycleOutcome = match run_cycle( - &sink, + &mut sink, &mut fleet, &mut writer, CycleFrame::new(CycleId(u64::from(c)), base_version), @@ -1280,23 +1305,24 @@ mod probe_ignition { } impl WalSink for FlakyWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { if self.fail_next.swap(false, Ordering::SeqCst) { - return Err(WriteFailed("G9 injected retryable WAL failure".into())); + return Err(CommitError::Io(WriteFailed( + "G9 injected retryable WAL failure".into(), + ))); } - self.inner.commit_cycle(base, batch).await + self.inner.commit_cycle(batch).await } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { - self.inner.scan_sealed(from_version).await + self.inner.scan_sealed(after_cycle).await } - async fn versions(&self) -> Result, WriteFailed> { - self.inner.versions().await + async fn timeline(&self) -> Result, WriteFailed> { + self.inner.timeline().await } } @@ -1317,7 +1343,7 @@ mod probe_ignition { ), ); - let sink = FlakyWal::new(); + let mut sink = FlakyWal::new(); let mut writer: BatchWriter> = BatchWriter::new(); let mut watermarks: HashMap> = HashMap::new(); @@ -1329,7 +1355,7 @@ mod probe_ignition { let before_phases = phase_cycle_snapshot(&fleet); sink.fail_next_commit(); let err = run_cycle( - &sink, + &mut sink, &mut fleet, &mut writer, CycleFrame::new(CycleId(1), DatasetVersion(0)), @@ -1357,7 +1383,7 @@ mod probe_ignition { let frozen_frame = failure.frame; let frozen_casts = failure.casts; - let sealed = seal_cycle(&sink, frozen_frame, frozen_casts) + let sealed = seal_cycle(&mut sink, frozen_frame, frozen_casts) .await .expect("G9: the retry with the frozen cast set must succeed"); assert_eq!( diff --git a/crates/lance-graph-supervisor/tests/probe_ignition_64k.rs b/crates/lance-graph-supervisor/tests/probe_ignition_64k.rs index a3c723348..2c48d1655 100644 --- a/crates/lance-graph-supervisor/tests/probe_ignition_64k.rs +++ b/crates/lance-graph-supervisor/tests/probe_ignition_64k.rs @@ -49,7 +49,8 @@ mod probe_ignition_64k { use lance_graph_planner::ir::Arena; use lance_graph_planner::owner_adapter::emit_bootstrap_intent; use lance_graph_planner::persist_sink::{ - CycleFrame, CycleId, DetachedCycleBatch, LandedSlot, SweepSlot, WalSink, WriteFailed, + CommitError, CommitOutcome, CycleFrame, CycleId, DetachedCycleBatch, FrameMeta, LandedSlot, + SweepSlot, WalSink, WriteFailed, }; use lance_graph_planner::strategy::style_strategy::StyleStrategy; use lance_graph_planner::traits::{ @@ -118,7 +119,11 @@ mod probe_ignition_64k { // MemWal, trimmed to what this probe asserts. ────────────────────────── struct SealedCycle { + frame: CycleFrame, version: DatasetVersion, + /// The batch's deterministic content hash — the reconciliation-first + /// idempotency key `commit_cycle` looks up BEFORE appending. + batch_hash: u64, landings: Vec, } @@ -150,56 +155,76 @@ mod probe_ignition_64k { impl WalSink for MemWal { async fn commit_cycle( - &self, - base: DatasetVersion, + &mut self, batch: DetachedCycleBatch, - ) -> Result { + ) -> Result { let mut sealed = self.sealed.lock().expect("MemWal poisoned"); + // Reconciliation-first: an already-durable (cycle, hash) is success, + // a matching cycle with a different hash fails closed. + if let Some(rec) = sealed.iter().find(|s| s.frame.cycle == batch.frame.cycle) { + return if rec.batch_hash == batch.batch_hash { + Ok(CommitOutcome::Reconciled { + current_head: rec.version, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash: rec.batch_hash, + offered_hash: batch.batch_hash, + }) + }; + } let head = sealed.last().map_or(DatasetVersion(0), |s| s.version); - if base != head { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is {head:?}" - ))); + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); } self.wal_writes.fetch_add(1, Ordering::SeqCst); let version = DatasetVersion(self.next_version.fetch_add(1, Ordering::SeqCst)); + let (cycle, batch_hash) = (batch.frame.cycle, batch.batch_hash); sealed.push(SealedCycle { + frame: batch.frame, version, + batch_hash, landings: batch.landings, }); - Ok(version) + Ok(CommitOutcome::Committed { + version, + cycle, + batch_hash, + }) } async fn scan_sealed( &self, - from_version: Option, + after_cycle: Option, ) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .filter(|s| from_version.is_none_or(|f| s.version > f)) + .filter(|s| after_cycle.is_none_or(|c| s.frame.cycle > c)) .flat_map(|s| { s.landings.iter().map(|slot| LandedSlot { - version: s.version, + cycle: s.frame.cycle, slot: slot.clone(), }) }) .collect()) } - async fn versions(&self) -> Result, WriteFailed> { + async fn timeline(&self) -> Result, WriteFailed> { Ok(self .sealed .lock() .expect("MemWal poisoned") .iter() - .map(|s| { - ( - s.landings.first().map_or(CycleId(0), |l| l.cycle), - s.version, - ) + .map(|s| FrameMeta { + cycle: s.frame.cycle, + base_version: s.frame.base_version, + batch_hash: s.batch_hash, }) .collect()) } @@ -281,7 +306,7 @@ mod probe_ignition_64k { ); } - let sink = MemWal::new(); + let mut sink = MemWal::new(); let mut writer: BatchWriter> = BatchWriter::new(); let mut watermarks: HashMap> = HashMap::new(); let position_base: u64 = 0; @@ -350,7 +375,7 @@ mod probe_ignition_64k { let wal_before = sink.wal_writes(); let base = sink.head(); let outcome: CycleOutcome = match run_cycle( - &sink, + &mut sink, &mut fleet, &mut writer, CycleFrame::new(CycleId(1), base), diff --git a/crates/lance-graph/src/graph/cycle_sink.rs b/crates/lance-graph/src/graph/cycle_sink.rs index 73ca28e9d..e711c923e 100644 --- a/crates/lance-graph/src/graph/cycle_sink.rs +++ b/crates/lance-graph/src/graph/cycle_sink.rs @@ -1,80 +1,88 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright The Lance Authors -//! The CONCRETE cognitive-cycle Lance sink — the storage-proven implementation of -//! `lance_graph_planner::persist_sink::WalSink` over the official Lance -//! transaction/write path (`lance = 9.0.0`). +//! The concrete Lance-backed cycle store — [`LanceCycleWriter`], the SOLE +//! application writer (Phase A of the canonical persistence contract, +//! operator-ruled 2026-08-09; supersedes the `LanceCycleSink` shipped in #911). //! -//! `persist_sink` (the planner seam) deliberately builds NO concrete sink: its -//! `FakeWalSink` proves the algebra (fence, ordering, coalescing, recovery) in -//! process memory only — "compile+test green ≠ storage proven" (the Ladybug -//! lesson). This module closes that gap: every trait operation here is true -//! against a REOPENED dataset on real storage. +//! # One logical writer, owned //! -//! # The §I.6 invariant (the ruled contract this sink makes physical) +//! There is exactly ONE logical application writer per cycle store. The 64k +//! thoughts / SoA owners are parallel PRODUCERS (fire-and-forget: they cast on +//! behalf of their mailbox and receive no acknowledgement), never Lance +//! writers. This type makes the topology structural: //! -//! ```text -//! 64k thoughts read sealed Vn → write-side temporal deinterlace → -//! one detached cycle batch → ONE official Lance commit → -//! exactly one real DatasetVersion Vn+1 → no open or partial visibility -//! ``` +//! - **non-`Clone`** — a second handle to the same writer cannot be minted; +//! - **`commit_cycle(&mut self, …)`** — two application commits cannot +//! interleave through the type boundary; +//! - **long-lived handle** — the writer OWNS its `Dataset` handle and current +//! head. Reads use the held handle; there is no per-operation reopen. The +//! dataset is opened exactly once at construction, and re-opened only to +//! resolve an ambiguous commit outcome (a lost acknowledgement). //! -//! - **One durable append per cycle.** `commit_cycle` performs a single -//! `Dataset::write` / `Dataset::append` (the official Lance insert path — -//! `InsertBuilder` under the hood, the same transaction machinery every Lance -//! writer uses). No bespoke ledger, no acknowledgement protocol, no parallel -//! replay system: Lance's own manifest/version chain IS the WAL. -//! - **The epistemic fence, both halves.** Pre-commit: the dataset's current -//! version must equal the cycle's sealed predecessor `base` (`Vn`), else the -//! commit is refused with nothing written. Post-commit: the published version -//! must be exactly `base + 1`. Lance's optimistic concurrency auto-resolves -//! append-append conflicts (a foreign interleaved writer would yield -//! `base + 2`), so under the one-writer-per-mailbox doctrine a post-check -//! mismatch is a LOUD timeline anomaly, never silently accepted. -//! - **Order is a write-side property.** Landings arrive already deinterlaced -//! (`DetachedCycleBatch::freeze` ran the loom); they are stored in that -//! canonical order and scanned back with Lance's in-order scan. This sink -//! never sorts on read. -//! - **All-or-nothing visibility.** An unsealed / fenced / failed cycle leaves -//! no rows and no version: after restart + reopen it is simply absent. -//! Recovery is a read of the sealed store (`scan_sealed` + the caller's -//! watermark in `recover_and_apply`) — idempotent without any sidecar state. +//! Lance's own transaction/manifest machinery (the backend durability path) +//! is INTERNAL to this one writer — `Dataset::write` / `Dataset::append` are +//! official atomic Lance MVCC commits, and nothing else writes here. An +//! unexpected head can only mean: an earlier commit became durable but its +//! response was lost; a restart reopened from a stale cached head; an +//! unauthorized writer violated the topology; or corruption. It is a +//! fence/reconciliation condition, never normal competition. //! -//! # Domain 0x09 — the patient SoA witness store (why the schema is rich) +//! # The governing storage rule (why there is no empty-cycle version) //! -//! The patient SoA at classid domain `0x09` is the ONLY place patient reasoning -//! is ever written to Lance. Everything else the reasoner touches — the -//! interlocked ontologies at domain `0x03`, crosswalks, RO edges — is IMMUTABLE -//! for the duration of a representation window: a cycle takes that immutability -//! for granted (its `base_version` names the sealed ontology-bearing predecessor -//! it read), and therefore never needs to restate ontology content. What it MUST -//! state — maximally richly — is the WITNESSING: +//! **No artifact-backed semantic change → no write → no new `DatasetVersion`.** +//! `persist_cycle` partitions intent-only casts out and returns +//! `CommitOutcome::NoChange` without ever calling this writer, so a timer +//! tick, an empty cycle, a `Continue`, a held intent or a pure kanban step +//! performs ZERO Lance operations here. The #911 deliberate empty-cycle +//! versioning is REMOVED. Kanban progress rides along ONLY when an artifact +//! commit happens anyway (the moves of artifact casts, sealed in the same +//! atomic commit). //! -//! - **`payload`** carries the canonical witness node bytes (the 512-byte -//! `key(16) | edges(16) | value(480)` node ABI): the EpisodicWitness row — -//! visited ontology addresses, executed crosswalk mappings, the exact RO / -//! ontology edge identifiers walked, supporting / contradicting / missing -//! observations, NARS truth + confidence, differential branches. Domain-0x09 -//! keys, edges pointing INTO the immutable 0x03 address space. -//! - **The landing columns** carry the dynamic-reasoning-update record: which -//! mailbox reasoned (`owner`), where in the canonical thought stream -//! (`stream_position`), which SoA row the update lands on (`row`), and the -//! Rubicon lifecycle step the thought cast (`move_*` — the sealed reflection -//! of the thinking, applied post-SEAL only). -//! - **The frame row** (one per cycle, `kind = 0`) seals the cycle ↔ version -//! mapping INSIDE the same atomic commit, so the coarse timeline -//! (`versions()`) survives restart with zero sidecar files — the sealed -//! versioning is literally a reflection of the thinking that produced it. +//! # No rollback, no compensating delete — reconciliation is authoritative //! -//! Downstream (the Gotham display, differential views, any consumer) reads the -//! sealed version — never a live recomputation: the witness is examined in -//! place, at the version its cycle published. +//! Lance 9 has no atomic expected-version fence for Append (the conflict +//! rebase runs even on a single-attempt commit; strict no-rebase mode exists +//! only for Overwrite — measured in `lance-9.0.0/src/io/commit.rs`), and a +//! published manifest is HISTORY (`Dataset::delete` creates another version; +//! it is not rollback — the #911 compensating delete is removed, not +//! repaired). Instead, idempotency is durable: every committed row carries +//! its `(cycle, batch_hash)` in the same commit, and [`WalSink::commit_cycle`] +//! reconciles FIRST — an already-durable batch returns +//! [`CommitOutcome::Reconciled`]; a matching cycle with a different hash +//! fails closed ([`CommitError::HashConflict`]); an append whose +//! acknowledgement was lost is resolved by re-submitting the SAME frozen +//! batch. Only when reconciliation itself cannot answer does +//! [`CommitError::Ambiguous`] surface. +//! +//! # Reference the new version; never reload normal state +//! +//! After a successful commit the caller already holds the outcome (version, +//! cycle, hash) and the submitted batch. The normal path performs **zero +//! reopens, zero scans, zero readbacks** — [`LanceCycleWriter::opens`] counts +//! every `Dataset::open` this writer ever performs so the invariant is +//! instrumented, not asserted. `scan_sealed` / `timeline` exist for recovery, +//! audit and downstream consumers, are bounded (`after_cycle` pushed into the +//! Lance scan as a predicate) and projected (the timeline never touches the +//! payload column). +//! +//! # Copy boundary (documented honestly, isolated for a later measured PR) +//! +//! This writer materializes the frozen batch's landings into Arrow builders +//! (one copy of each payload) and `scan_*` reads copy bytes back out +//! (`to_vec`). True zero-copy (Arc-backed Arrow buffers pinned over SoA +//! ranges) does not fit this focused repair; the copy boundary is exactly +//! these two seams and nothing else. The 512-byte witness ABI is enforced on +//! ARTIFACT payloads only — intent-only casts never reach this writer, so the +//! `restage_held` empty-payload shape can never trip the gate. +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use arrow_array::{ - builder::{BinaryBuilder, UInt32Builder, UInt8Builder}, - Array, BinaryArray, RecordBatch, RecordBatchIterator, UInt32Array, UInt64Array, UInt8Array, + builder::{FixedSizeBinaryBuilder, UInt32Builder, UInt8Builder}, + Array, FixedSizeBinaryArray, RecordBatch, RecordBatchIterator, UInt32Array, UInt64Array, + UInt8Array, }; use arrow_schema::{DataType, Field, Schema, SchemaRef}; use futures::TryStreamExt; @@ -84,54 +92,41 @@ use lance_graph_contract::collapse_gate::MailboxId; use lance_graph_contract::kanban::{ExecTarget, KanbanColumn, KanbanMove}; use lance_graph_contract::scheduler::DatasetVersion; use lance_graph_planner::persist_sink::{ - CycleId, DetachedCycleBatch, LandedSlot, SweepSlot, WalSink, WriteFailed, + CommitError, CommitOutcome, CycleId, DetachedCycleBatch, FrameMeta, LandedSlot, SweepSlot, + WalSink, WriteFailed, }; -/// Row kind discriminant: the per-cycle frame row (cycle ↔ version mapping, -/// sealed inside the same atomic commit as its landings). +/// Row kind: the per-cycle frame row (cycle identity + batch hash + read +/// horizon — compact metadata, sealed atomically with its landings). const KIND_FRAME: u8 = 0; -/// Row kind discriminant: one landing (a thought's persistence record). +/// Row kind: one artifact landing's TRANSITION METADATA (move + position; +/// payload column NULL — compact, per cast). const KIND_LANDING: u8 = 1; -/// Row kind discriminant: one coalesced-image row — the FINAL payload of a -/// dirty SoA row after the per-row fold (`DetachedCycleBatch::image`), made -/// durable in the same atomic commit so the store carries the coherent cycle -/// image itself, not just the per-cast history it folds from. +/// Row kind: one coalesced-image row — `row` + the FINAL 512-byte payload +/// after the per-row fold. Exactly one payload per dirty row per cycle: 64 +/// same-row breaths durably cost ONE image row, never 64 × 512 bytes. const KIND_IMAGE: u8 = 2; -/// The canonical witness-node payload size: `key(16) | edges(16) | value(480)` -/// — the 512-byte node row stride with its 16-byte edge reservation. Every -/// landing / image payload persisted by this sink MUST be exactly this long; -/// a malformed witness row is refused before anything durable happens. +/// The canonical witness-node ABI for ARTIFACT payloads: +/// `key(16) | edges(16) | value(480)` — the 512-byte node row stride. pub const EPISODIC_WITNESS_BYTES: usize = 512; -/// The Arrow schema of the cycle store — one dataset, three row kinds: the -/// per-cycle frame row (`kind = 0`), the per-cast landing rows (`kind = 1`, -/// the table below), and the coalesced-image rows (`kind = 2`: `row` + the -/// FINAL 512-byte payload after the per-row fold, `stream_position`/`owner` -/// zero, moves null — the durable coherent cycle image). -/// -/// | column | type | frame row | landing row | -/// |-------------------------------|-----------------|-----------|-------------| -/// | `kind` | `UInt8` | 0 | 1 | -/// | `cycle` | `UInt64` | cycle id | cycle id | -/// | `base_version` | `UInt64` | `Vn` | `Vn` | -/// | `stream_position` | `UInt64` | 0 | canonical order key | -/// | `owner` | `UInt32` | 0 | mailbox | -/// | `row` | `UInt64` | 0 | SoA row | -/// | `move_mailbox` | `UInt32?` | null | paired move (or null) | -/// | `move_from` / `move_to` | `UInt8?` | null | Rubicon edge | -/// | `move_witness_chain_position` | `UInt32?` | null | witness pointer (R4) | -/// | `move_exec` | `UInt8?` | null | exec target | -/// | `payload` | `Binary` | empty | witness node bytes | +/// The Arrow schema — flat rows, three kinds, payload physically +/// `FixedSizeBinary(512)` (nullable: only image rows carry it). /// -/// The sealed version of every row's cycle is `base_version + 1` — an identity -/// the commit path VERIFIES against the real published Lance version (it is -/// never assumed), so reads may derive it without a sidecar mapping. +/// | column | frame | landing | image | +/// |---|---|---|---| +/// | `kind` | 0 | 1 | 2 | +/// | `cycle` / `base_version` / `batch_hash` | ✓ | ✓ | ✓ | +/// | `stream_position` / `owner` / `row` | 0 | ✓ | winner's / 0 / row | +/// | `move_*` (nullable) | null | cast's move | null | +/// | `payload` (`FixedSizeBinary(512)`, nullable) | null | null | final image | pub fn cycle_store_schema() -> SchemaRef { Arc::new(Schema::new(vec![ Field::new("kind", DataType::UInt8, false), Field::new("cycle", DataType::UInt64, false), Field::new("base_version", DataType::UInt64, false), + Field::new("batch_hash", DataType::UInt64, false), Field::new("stream_position", DataType::UInt64, false), Field::new("owner", DataType::UInt32, false), Field::new("row", DataType::UInt64, false), @@ -140,63 +135,162 @@ pub fn cycle_store_schema() -> SchemaRef { Field::new("move_to", DataType::UInt8, true), Field::new("move_witness_chain_position", DataType::UInt32, true), Field::new("move_exec", DataType::UInt8, true), - Field::new("payload", DataType::Binary, false), + Field::new( + "payload", + DataType::FixedSizeBinary(EPISODIC_WITNESS_BYTES as i32), + true, + ), ])) } -/// The concrete Lance-backed cycle sink. +/// The sole owned application writer over one Lance cycle store. /// -/// Cheap to clone / recreate: it holds only the dataset path and opens the -/// dataset per operation (the restart-survival guarantee is thereby exercised on -/// EVERY call, not just in tests). Point it at the domain-0x09 patient witness -/// store (e.g. `/witness_cycles.lance`) — one sink instance per store. -#[derive(Debug, Clone)] -pub struct LanceCycleSink { +/// Deliberately **non-`Clone`**: constructing a second writer over the same +/// path is a topology violation the type cannot prevent across processes, but +/// within a process the exclusive `&mut` commit boundary plus non-cloneability +/// make interleaved application commits unrepresentable. +#[derive(Debug)] +pub struct LanceCycleWriter { dataset_path: String, + /// The long-lived handle. `None` until the first committed cycle creates + /// the dataset (an empty store is a state, not an error). Once `Some`, it + /// NEVER degrades back to `None` — a store that existed cannot become + /// "empty" again (see [`reopen`](Self::reopen)). + ds: Option, + /// `Dataset::open` count — startup + ambiguity-resolution ONLY. The + /// normal-path invariant (zero post-success reopens) is instrumented here. + opens: AtomicU64, + /// Reconciliation-scan count ([`find_frame`](Self::find_frame) calls). The + /// NORMAL commit path performs ZERO of these: a fresh monotonic cycle + /// appends directly over the in-memory head + cycle watermark; the scan + /// runs only on a fence mismatch, a `cycle ≤ watermark` re-submission, or + /// ambiguity resolution. Instrumented so "zero scans on the normal path" + /// is measured, not asserted. + reconcile_scans: AtomicU64, + /// The highest cycle known durable in THIS store (seeded at open from one + /// bounded frame scan; advanced in memory on every commit). Monotonic — + /// `cycle > committed_through` proves the cycle cannot already be durable, + /// which is what makes the scan-free fast path sound. + committed_through: Option, } -impl LanceCycleSink { - /// A sink over the Lance dataset at `path` (local path or object-store URI — - /// anything `Dataset::open` accepts). The dataset is created on the first - /// committed cycle; a missing dataset is simply "nothing sealed yet". - #[must_use] - pub fn new(path: impl Into) -> Self { - Self { - dataset_path: path.into(), +/// The process-local single-writer registry: one LIVE [`LanceCycleWriter`] per +/// dataset path. `non-Clone + &mut self` serializes commits on one instance; +/// this registry closes the remaining in-process hole (a second `open` of the +/// same path is REFUSED while the first writer lives). Cross-PROCESS +/// exclusivity remains a deployment lease this crate cannot enforce — stated, +/// not implied away. +static OPEN_WRITERS: std::sync::LazyLock>> = + std::sync::LazyLock::new(|| std::sync::Mutex::new(std::collections::HashSet::new())); + +impl Drop for LanceCycleWriter { + fn drop(&mut self) { + if let Ok(mut set) = OPEN_WRITERS.lock() { + set.remove(&self.dataset_path); + } + } +} + +impl LanceCycleWriter { + /// Open the writer over `path` (local path or object-store URI). Performs + /// the ONE startup open (plus one bounded, frame-projected seed scan when + /// the store exists); a missing dataset is an empty store. + /// + /// Refuses a second live writer on the same path in this process (the + /// one-logical-writer topology, enforced rather than narrated). Refuses a + /// store whose schema is not this writer's layout — a pre-Phase-A (#911) + /// store is REJECTED loudly, never silently reinterpreted. + pub async fn open(path: impl Into) -> Result { + let dataset_path = path.into(); + { + let mut set = OPEN_WRITERS + .lock() + .map_err(|_| WriteFailed("writer registry poisoned".into()))?; + if !set.insert(dataset_path.clone()) { + return Err(WriteFailed(format!( + "a live LanceCycleWriter already owns {dataset_path} in this process — \ + one logical writer per store (drop it first)" + ))); + } } + let opens = AtomicU64::new(0); + let ds = match Dataset::open(&dataset_path).await { + Ok(ds) => { + opens.fetch_add(1, Ordering::Relaxed); + let expected = cycle_store_schema(); + let got = ds.schema(); + for field in expected.fields() { + if got.field(field.name()).is_none() { + OPEN_WRITERS + .lock() + .ok() + .map(|mut s| s.remove(&dataset_path)); + return Err(WriteFailed(format!( + "store at {dataset_path} is missing column `{}` — not this \ + writer's layout (a pre-Phase-A store is rejected, not \ + reinterpreted; migrate or discard it explicitly)", + field.name() + ))); + } + } + Some(ds) + } + Err(lance::Error::DatasetNotFound { .. }) => None, + Err(e) => { + OPEN_WRITERS + .lock() + .ok() + .map(|mut s| s.remove(&dataset_path)); + return Err(WriteFailed(format!("open {dataset_path}: {e}"))); + } + }; + let mut w = Self { + dataset_path, + ds, + opens, + reconcile_scans: AtomicU64::new(0), + committed_through: None, + }; + if w.ds.is_some() { + // One bounded, projected seed read: the highest durable cycle. + // This is startup hydration (allowed), not a normal-path scan. + let frames = w.timeline().await?; + w.committed_through = frames.iter().map(|f| f.cycle).max(); + } + Ok(w) + } + + /// The store's current head version (`0` = empty store) — the in-memory + /// token the normal path references instead of reloading state. + #[must_use] + pub fn head(&self) -> DatasetVersion { + DatasetVersion(self.ds.as_ref().map_or(0, |d| d.version().version)) } - /// The dataset path this sink commits to. + /// The dataset path this writer commits to. #[must_use] pub fn dataset_path(&self) -> &str { &self.dataset_path } - /// Open the store if it exists; `None` = nothing sealed yet (a state, not an - /// error — distinguishing it from a real I/O failure is the caller-visible - /// difference between an empty timeline and a broken one). - async fn open_if_exists(&self) -> Result, WriteFailed> { - match Dataset::open(&self.dataset_path).await { - Ok(ds) => Ok(Some(ds)), - Err(lance::Error::DatasetNotFound { .. }) => Ok(None), - Err(e) => Err(WriteFailed(format!("open {}: {e}", self.dataset_path))), - } + /// How many `Dataset::open` calls this writer has EVER performed — + /// startup (≤1) plus ambiguity resolutions. The zero-reload falsifier + /// asserts this stays flat across normal commits and reads. + #[must_use] + pub fn opens(&self) -> u64 { + self.opens.load(Ordering::Relaxed) } - /// Build the single atomic RecordBatch for a cycle: the frame row first, - /// then the landings in their ALREADY-canonical order (the loom ran in - /// `DetachedCycleBatch::freeze`; storage order = stream order by contract), - /// then the coalesced-image rows (`row → final payload`) so the coherent - /// cycle image is durable alongside the per-cast history it folds from. - /// - /// Every landing / image payload must be exactly [`EPISODIC_WITNESS_BYTES`] - /// — the canonical 512-byte node row — or the whole cycle is refused - /// before anything durable happens. - fn build_batch(batch: &DetachedCycleBatch) -> Result { - let n = batch.landings.len() + batch.image.len() + 1; + /// Build the single atomic RecordBatch: frame row, landing-metadata rows + /// (canonical order, payload null), image rows (row-ascending, final + /// payload). Refuses a non-512-byte ARTIFACT payload before anything + /// durable happens (intent-only casts never reach this writer). + fn build_batch(batch: &DetachedCycleBatch) -> Result { + let n = 1 + batch.landings.len() + batch.image.len(); let mut kind = Vec::with_capacity(n); let mut cycle = Vec::with_capacity(n); let mut base_version = Vec::with_capacity(n); + let mut batch_hash = Vec::with_capacity(n); let mut stream_position = Vec::with_capacity(n); let mut owner = Vec::with_capacity(n); let mut row = Vec::with_capacity(n); @@ -205,37 +299,37 @@ impl LanceCycleSink { let mut move_to = UInt8Builder::with_capacity(n); let mut move_wcp = UInt32Builder::with_capacity(n); let mut move_exec = UInt8Builder::with_capacity(n); - let mut payload = BinaryBuilder::new(); - - // Frame row — the cycle ↔ version mapping, sealed atomically with its - // landings (a zero-landing cycle still advances the timeline). - kind.push(KIND_FRAME); - cycle.push(batch.frame.cycle.0); - base_version.push(batch.frame.base_version.0); - stream_position.push(0); - owner.push(0); - row.push(0); + let mut payload = FixedSizeBinaryBuilder::with_capacity(n, EPISODIC_WITNESS_BYTES as i32); + + let mut push_common = |k: u8, sp: u64, ow: u32, rw: u64| { + kind.push(k); + cycle.push(batch.frame.cycle.0); + base_version.push(batch.frame.base_version.0); + batch_hash.push(batch.batch_hash); + stream_position.push(sp); + owner.push(ow); + row.push(rw); + }; + + // Frame row — compact metadata, no move, no payload. + push_common(KIND_FRAME, 0, 0, 0); move_mailbox.append_null(); move_from.append_null(); move_to.append_null(); move_wcp.append_null(); move_exec.append_null(); - payload.append_value([]); + payload.append_null(); + // Landing metadata rows — the sparse transition set, payload NULL. for s in &batch.landings { if s.payload.len() != EPISODIC_WITNESS_BYTES { - return Err(WriteFailed(format!( - "landing payload for row {} is {} bytes, expected the canonical {EPISODIC_WITNESS_BYTES}", + return Err(CommitError::Io(WriteFailed(format!( + "artifact payload for row {} is {} bytes, ABI requires {EPISODIC_WITNESS_BYTES}", s.row, s.payload.len() - ))); + )))); } - kind.push(KIND_LANDING); - cycle.push(s.cycle.0); - base_version.push(batch.frame.base_version.0); - stream_position.push(s.stream_position); - owner.push(s.owner); - row.push(s.row); + push_common(KIND_LANDING, s.stream_position, s.owner, s.row); match &s.paired_move { Some(m) => { move_mailbox.append_value(m.mailbox); @@ -252,33 +346,20 @@ impl LanceCycleSink { move_exec.append_null(); } } - payload.append_value(&s.payload); + payload.append_null(); } - // Coalesced-image rows: the final per-row state after the stream-order - // fold. `BTreeMap` iteration gives a deterministic (row-ascending) - // stored order. Same-cycle landings already passed the 512-byte gate, - // and the image is a fold over exactly those payloads — the length - // check here guards the invariant independently rather than assuming it. + // Image rows — the coalesced final payload, once per dirty row. for (row_id, image_payload) in &batch.image { - if image_payload.len() != EPISODIC_WITNESS_BYTES { - return Err(WriteFailed(format!( - "image payload for row {row_id} is {} bytes, expected the canonical {EPISODIC_WITNESS_BYTES}", - image_payload.len() - ))); - } - kind.push(KIND_IMAGE); - cycle.push(batch.frame.cycle.0); - base_version.push(batch.frame.base_version.0); - stream_position.push(0); - owner.push(0); - row.push(*row_id); + push_common(KIND_IMAGE, 0, 0, *row_id); move_mailbox.append_null(); move_from.append_null(); move_to.append_null(); move_wcp.append_null(); move_exec.append_null(); - payload.append_value(image_payload); + payload + .append_value(image_payload) + .map_err(|e| CommitError::Io(WriteFailed(format!("image row {row_id}: {e}"))))?; } RecordBatch::try_new( @@ -287,6 +368,7 @@ impl LanceCycleSink { Arc::new(UInt8Array::from(kind)), Arc::new(UInt64Array::from(cycle)), Arc::new(UInt64Array::from(base_version)), + Arc::new(UInt64Array::from(batch_hash)), Arc::new(UInt64Array::from(stream_position)), Arc::new(UInt32Array::from(owner)), Arc::new(UInt64Array::from(row)), @@ -298,33 +380,269 @@ impl LanceCycleSink { Arc::new(payload.finish()), ], ) - .map_err(|e| WriteFailed(format!("build cycle batch: {e}"))) + .map_err(|e| CommitError::Io(WriteFailed(format!("build cycle batch: {e}")))) } - /// Read the store's rows of ONE kind at its LATEST version, in stored - /// (insertion) order — Lance's in-order scan; this sink never sorts on - /// read. The kind predicate is pushed into the scan so frame/image reads - /// never materialize landing payloads (and vice versa). - async fn read_rows_of_kind( + /// Look this cycle's durable frame up (projected `cycle` + `batch_hash` + /// under a `kind = 0 AND cycle = …` predicate) — the reconciliation read. + async fn find_frame(&self, cycle: CycleId) -> Result, WriteFailed> { + let Some(ds) = self.ds.as_ref() else { + return Ok(None); + }; + let mut scan = ds.scan(); + scan.filter(&format!("kind = {KIND_FRAME} AND cycle = {}", cycle.0)) + .map_err(|e| WriteFailed(format!("filter: {e}")))?; + scan.project(&["batch_hash"]) + .map_err(|e| WriteFailed(format!("project: {e}")))?; + let batches: Vec = scan + .try_into_stream() + .await + .map_err(|e| WriteFailed(format!("scan: {e}")))? + .try_collect() + .await + .map_err(|e| WriteFailed(format!("collect: {e}")))?; + for b in &batches { + let h: &UInt64Array = b + .column_by_name("batch_hash") + .and_then(|c| c.as_any().downcast_ref()) + .ok_or_else(|| WriteFailed("missing column batch_hash".into()))?; + if b.num_rows() > 0 { + return Ok(Some(h.value(0))); + } + } + Ok(None) + } + + /// Re-open the dataset from storage — ambiguity resolution ONLY (counted). + /// + /// A store that existed can NEVER degrade to "empty": if this writer holds + /// a handle and the reopen reports `DatasetNotFound` (transient listing + /// failure, eventual consistency, or genuine corruption), the OLD handle + /// is kept and an error is returned — the caller's outcome stays + /// `Ambiguous`, and the next commit can never fall into `Create` over a + /// store that has history. + async fn reopen(&mut self) -> Result<(), WriteFailed> { + match Dataset::open(&self.dataset_path).await { + Ok(ds) => { + self.opens.fetch_add(1, Ordering::Relaxed); + self.ds = Some(ds); + Ok(()) + } + Err(lance::Error::DatasetNotFound { .. }) => { + self.opens.fetch_add(1, Ordering::Relaxed); + if self.ds.is_some() { + return Err(WriteFailed(format!( + "reopen {}: store reported NOT FOUND but this writer holds \ + history — keeping the existing handle (a store never \ + becomes empty again); treat the outcome as ambiguous", + self.dataset_path + ))); + } + Ok(()) + } + Err(e) => Err(WriteFailed(format!("reopen {}: {e}", self.dataset_path))), + } + } + + /// How many reconciliation scans ([`find_frame`](Self::find_frame)) this + /// writer has EVER run. Zero across a run of fresh monotonic commits — + /// the "zero scans on the normal path" falsifier reads this. + #[must_use] + pub fn reconcile_scans(&self) -> u64 { + self.reconcile_scans.load(Ordering::Relaxed) + } +} + +impl WalSink for LanceCycleWriter { + /// THE single durable commit for a whole cycle — reconciliation-first, + /// fence second, append third; the outcome is fully honored. + async fn commit_cycle( + &mut self, + batch: DetachedCycleBatch, + ) -> Result { + // Zero-artifact batches never reach a sink (persist_cycle partitions), + // but the invariant is enforced here too — this writer NEVER creates a + // version for nothing. + if batch.landings.is_empty() { + return Ok(CommitOutcome::NoChange { + head: DatasetVersion(batch.frame.base_version.0), + }); + } + // 1. The scan-free FAST PATH decision. A fresh monotonic cycle + // (`cycle > committed_through`, seeded at open) provably cannot be + // durable yet, and a matching fence proves the horizon — so the + // normal path appends DIRECTLY, zero reads. Reconciliation runs + // only when something is off: a fence mismatch (lost-response + // restart / stale cache / topology violation) or a re-submission + // at-or-below the durable cycle watermark. + let head = self.head(); + let fresh = self + .committed_through + .is_none_or(|ct| batch.frame.cycle > ct); + if !fresh || batch.frame.base_version != head { + // Reconciliation: an already-durable (cycle, hash) is success; a + // matching cycle with a different hash fails closed; a genuinely + // absent cycle with a bad fence is Fenced (nothing written). + self.reconcile_scans.fetch_add(1, Ordering::Relaxed); + match self.find_frame(batch.frame.cycle).await { + Ok(Some(stored_hash)) => { + return if stored_hash == batch.batch_hash { + self.committed_through = Some( + self.committed_through + .map_or(batch.frame.cycle, |ct| ct.max(batch.frame.cycle)), + ); + Ok(CommitOutcome::Reconciled { + current_head: head, + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } else { + Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash, + offered_hash: batch.batch_hash, + }) + }; + } + Ok(None) => { + if batch.frame.base_version != head { + return Err(CommitError::Fenced { current_head: head }); + } + // cycle ≤ watermark but absent (a gap id) with a good + // fence: legitimate — fall through to the append. + } + Err(e) => { + // The reconciliation read itself failed — nothing written + // yet: safe to report as refused I/O. + return Err(CommitError::Io(e)); + } + } + } + // 3. The single atomic Lance MVCC commit. + let record_batch = Self::build_batch(&batch)?; + let schema = cycle_store_schema(); + let reader = RecordBatchIterator::new(vec![Ok(record_batch)], schema); + let append_result = match self.ds.as_mut() { + None => match Dataset::write( + reader, + &self.dataset_path, + Some(WriteParams { + mode: WriteMode::Create, + ..Default::default() + }), + ) + .await + { + Ok(ds) => { + self.ds = Some(ds); + Ok(()) + } + Err(e) => Err(e), + }, + Some(ds) => ds.append(reader, None).await, + }; + match append_result { + Ok(()) => { + self.committed_through = Some( + self.committed_through + .map_or(batch.frame.cycle, |ct| ct.max(batch.frame.cycle)), + ); + Ok(CommitOutcome::Committed { + // The ACTUAL publication version returned by the commit — + // accepted as-is, never "corrected" (no rollback, no + // delete, no derived identity). + version: self.head(), + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } + Err(e) => { + // The commit's outcome is UNKNOWN (the manifest may or may not + // have published before the failure). Reconcile from storage: + // reopen (counted; NEVER degrades an existing handle), then + // look for our durable identity. + let cause = e.to_string(); + if let Err(re) = self.reopen().await { + return Err(CommitError::Ambiguous { + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + cause: format!("append failed ({cause}); reopen failed ({re})"), + }); + } + self.reconcile_scans.fetch_add(1, Ordering::Relaxed); + match self.find_frame(batch.frame.cycle).await { + Ok(Some(stored_hash)) if stored_hash == batch.batch_hash => { + self.committed_through = Some( + self.committed_through + .map_or(batch.frame.cycle, |ct| ct.max(batch.frame.cycle)), + ); + Ok(CommitOutcome::Reconciled { + current_head: self.head(), + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + }) + } + Ok(Some(stored_hash)) => Err(CommitError::HashConflict { + cycle: batch.frame.cycle, + stored_hash, + offered_hash: batch.batch_hash, + }), + // Proven absent: nothing landed — safe to regenerate. + Ok(None) => Err(CommitError::Io(WriteFailed(format!( + "append failed with nothing published: {cause}" + )))), + Err(re) => Err(CommitError::Ambiguous { + cycle: batch.frame.cycle, + batch_hash: batch.batch_hash, + cause: format!("append failed ({cause}); reconciliation failed ({re})"), + }), + } + } + } + } + + /// Committed landing METADATA in stored canonical order, bounded by + /// `after_cycle` (pushed into the Lance scan). Payloads are NOT read here + /// — landing rows carry none (the durable payloads live in the coalesced + /// image, read via [`LanceCycleWriter::scan_image`]); returned slots carry + /// empty payload vectors. + async fn scan_sealed( &self, - ds: &Dataset, - kind_filter: u8, - ) -> Result, WriteFailed> { + after_cycle: Option, + ) -> Result, WriteFailed> { + let Some(ds) = self.ds.as_ref() else { + return Ok(Vec::new()); + }; let mut scan = ds.scan(); scan.scan_in_order(true); - scan.filter(&format!("kind = {kind_filter}")) - .map_err(|e| WriteFailed(format!("filter {}: {e}", self.dataset_path)))?; + let filter = match after_cycle { + Some(c) => format!("kind = {KIND_LANDING} AND cycle > {}", c.0), + None => format!("kind = {KIND_LANDING}"), + }; + scan.filter(&filter) + .map_err(|e| WriteFailed(format!("filter: {e}")))?; + scan.project(&[ + "cycle", + "stream_position", + "owner", + "row", + "move_mailbox", + "move_from", + "move_to", + "move_witness_chain_position", + "move_exec", + ]) + .map_err(|e| WriteFailed(format!("project: {e}")))?; let batches: Vec = scan .try_into_stream() .await - .map_err(|e| WriteFailed(format!("scan {}: {e}", self.dataset_path)))? + .map_err(|e| WriteFailed(format!("scan: {e}")))? .try_collect() .await - .map_err(|e| WriteFailed(format!("collect {}: {e}", self.dataset_path)))?; - - let mut rows = Vec::new(); + .map_err(|e| WriteFailed(format!("collect: {e}")))?; + let mut out = Vec::new(); for b in &batches { - let col_u8 = |name: &str| -> Result<&UInt8Array, WriteFailed> { + let col_u64 = |name: &str| -> Result<&UInt64Array, WriteFailed> { b.column_by_name(name) .and_then(|c| c.as_any().downcast_ref()) .ok_or_else(|| WriteFailed(format!("missing column {name}"))) @@ -334,14 +652,12 @@ impl LanceCycleSink { .and_then(|c| c.as_any().downcast_ref()) .ok_or_else(|| WriteFailed(format!("missing column {name}"))) }; - let col_u64 = |name: &str| -> Result<&UInt64Array, WriteFailed> { + let col_u8 = |name: &str| -> Result<&UInt8Array, WriteFailed> { b.column_by_name(name) .and_then(|c| c.as_any().downcast_ref()) .ok_or_else(|| WriteFailed(format!("missing column {name}"))) }; - let kind = col_u8("kind")?; let cycle = col_u64("cycle")?; - let base_version = col_u64("base_version")?; let stream_position = col_u64("stream_position")?; let owner = col_u32("owner")?; let row = col_u64("row")?; @@ -350,11 +666,6 @@ impl LanceCycleSink { let move_to = col_u8("move_to")?; let move_wcp = col_u32("move_witness_chain_position")?; let move_exec = col_u8("move_exec")?; - let payload: &BinaryArray = b - .column_by_name("payload") - .and_then(|c| c.as_any().downcast_ref()) - .ok_or_else(|| WriteFailed("missing column payload".into()))?; - for i in 0..b.num_rows() { let paired_move = if move_mailbox.is_valid(i) { Some(KanbanMove { @@ -367,249 +678,102 @@ impl LanceCycleSink { } else { None }; - rows.push(StoredRow { - kind: kind.value(i), + out.push(LandedSlot { cycle: CycleId(cycle.value(i)), - base_version: DatasetVersion(base_version.value(i)), slot: SweepSlot { cycle: CycleId(cycle.value(i)), stream_position: stream_position.value(i), owner: owner.value(i), row: row.value(i), paired_move, - payload: payload.value(i).to_vec(), + payload: Vec::new(), }, }); } } - Ok(rows) - } -} - -/// One decoded store row (frame or landing) — internal read shape. -struct StoredRow { - kind: u8, - cycle: CycleId, - base_version: DatasetVersion, - slot: SweepSlot, -} - -impl StoredRow { - /// The version this row's cycle sealed into — `base + 1`, the identity the - /// commit path verified against the real published Lance version. - fn sealed_version(&self) -> DatasetVersion { - DatasetVersion(self.base_version.0 + 1) + Ok(out) } -} -impl LanceCycleSink { - /// Read a sealed cycle's durable coalesced image: `row → final payload` - /// after the write-side per-row fold. This is the coherent end-state a - /// downstream consumer (a view, the Gotham display) reads — the per-cast - /// history behind it stays available via [`WalSink::scan_sealed`]. Projects - /// `row` + `payload` under a `kind = 2 AND cycle = …` predicate. An empty - /// map = the cycle is unknown or landed nothing. - pub async fn scan_image( - &self, - cycle: CycleId, - ) -> Result>, WriteFailed> { - let Some(ds) = self.open_if_exists().await? else { - return Ok(std::collections::BTreeMap::new()); + /// The coarse timeline — frame rows only, projected `cycle` + + /// `base_version` + `batch_hash`: the payload column is never scanned. + async fn timeline(&self) -> Result, WriteFailed> { + let Some(ds) = self.ds.as_ref() else { + return Ok(Vec::new()); }; let mut scan = ds.scan(); scan.scan_in_order(true); - scan.filter(&format!("kind = {KIND_IMAGE} AND cycle = {}", cycle.0)) - .map_err(|e| WriteFailed(format!("filter {}: {e}", self.dataset_path)))?; - scan.project(&["row", "payload"]) - .map_err(|e| WriteFailed(format!("project {}: {e}", self.dataset_path)))?; + scan.filter(&format!("kind = {KIND_FRAME}")) + .map_err(|e| WriteFailed(format!("filter: {e}")))?; + scan.project(&["cycle", "base_version", "batch_hash"]) + .map_err(|e| WriteFailed(format!("project: {e}")))?; let batches: Vec = scan .try_into_stream() .await - .map_err(|e| WriteFailed(format!("scan {}: {e}", self.dataset_path)))? + .map_err(|e| WriteFailed(format!("scan: {e}")))? .try_collect() .await - .map_err(|e| WriteFailed(format!("collect {}: {e}", self.dataset_path)))?; - let mut out = std::collections::BTreeMap::new(); + .map_err(|e| WriteFailed(format!("collect: {e}")))?; + let mut out = Vec::new(); for b in &batches { - let row: &UInt64Array = b - .column_by_name("row") + let cycle: &UInt64Array = b + .column_by_name("cycle") .and_then(|c| c.as_any().downcast_ref()) - .ok_or_else(|| WriteFailed("missing column row".into()))?; - let payload: &BinaryArray = b - .column_by_name("payload") + .ok_or_else(|| WriteFailed("missing column cycle".into()))?; + let base: &UInt64Array = b + .column_by_name("base_version") .and_then(|c| c.as_any().downcast_ref()) - .ok_or_else(|| WriteFailed("missing column payload".into()))?; + .ok_or_else(|| WriteFailed("missing column base_version".into()))?; + let hash: &UInt64Array = b + .column_by_name("batch_hash") + .and_then(|c| c.as_any().downcast_ref()) + .ok_or_else(|| WriteFailed("missing column batch_hash".into()))?; for i in 0..b.num_rows() { - out.insert(row.value(i), payload.value(i).to_vec()); + out.push(FrameMeta { + cycle: CycleId(cycle.value(i)), + base_version: DatasetVersion(base.value(i)), + batch_hash: hash.value(i), + }); } } Ok(out) } } -impl WalSink for LanceCycleSink { - /// THE single amortized durable append for a whole cycle, over the official - /// Lance insert path — one commit, one new `DatasetVersion`, all-or-nothing. - /// - /// The epistemic fence, both halves: - /// 1. **Pre-commit:** the store's current version must equal `base` (`Vn`). - /// An empty store has head `DatasetVersion(0)`, so the first cycle must - /// declare base 0 (it read no sealed predecessor). A stale base is - /// refused with NOTHING written. - /// 2. **Post-commit:** the published version must be exactly `base + 1`. - /// Lance auto-resolves append-append conflicts (there is no - /// expected-version conditional append in the official API — the rebase - /// runs even on a single-attempt commit for Append operations), so a - /// foreign interleaved writer can land this batch at `base + 2`. When - /// that is detected, the fence is made EFFECTIVE retroactively: the - /// just-published cycle rows are removed again with an official - /// `Dataset::delete` scoped to exactly this cycle's rows, and only THEN - /// is the retryable [`WriteFailed`] returned — so "write failed" is - /// true at the visible head (nothing of this cycle remains readable), - /// the driver's regenerate-from-`Vn` contract stays sound, and no rows - /// survive under a shifted `sealed_version` identity. If the - /// compensating delete itself fails, the error says so explicitly and - /// names the orphaned version — the one manual-reconciliation corner, - /// reachable only when the one-writer §I.6 doctrine was already - /// violated by a foreign writer. - async fn commit_cycle( - &self, - base: DatasetVersion, - batch: DetachedCycleBatch, - ) -> Result { - if batch.frame.base_version != base { - return Err(WriteFailed(format!( - "frame base {:?} != commit base {base:?}", - batch.frame.base_version - ))); - } - let record_batch = Self::build_batch(&batch)?; - let schema = cycle_store_schema(); - let published = match self.open_if_exists().await? { - None => { - // Empty store: sealed head is DatasetVersion(0) by convention. - if base.0 != 0 { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is DatasetVersion(0) (empty store)" - ))); - } - let reader = RecordBatchIterator::new(vec![Ok(record_batch)], schema); - let params = WriteParams { - mode: WriteMode::Create, - ..Default::default() - }; - let ds = Dataset::write(reader, &self.dataset_path, Some(params)) - .await - .map_err(|e| WriteFailed(format!("create commit: {e}")))?; - ds.version().version - } - Some(mut ds) => { - let head = ds.version().version; - if head != base.0 { - return Err(WriteFailed(format!( - "stale base {base:?}: sealed head is DatasetVersion({head})" - ))); - } - let reader = RecordBatchIterator::new(vec![Ok(record_batch)], schema); - ds.append(reader, None) - .await - .map_err(|e| WriteFailed(format!("append commit: {e}")))?; - let published = ds.version().version; - if published != base.0 + 1 { - // A foreign writer interleaved between the fence check and - // the commit; Lance's append rebase landed this batch at a - // shifted version. Make the fence effective retroactively: - // remove exactly this cycle's just-appended rows, then - // report the (now-true) retryable failure. The cycle id is - // an unsealed identity at this point — no earlier sealed - // rows can carry it — so the predicate is exact. - let compensate = ds - .delete(&format!( - "cycle = {} AND base_version = {}", - batch.frame.cycle.0, base.0 - )) - .await; - return Err(match compensate { - Ok(_) => WriteFailed(format!( - "fenced post-publication: a foreign writer moved the head past \ - {base:?} (batch landed at DatasetVersion({published})); the \ - cycle's rows were deleted again — nothing of cycle {} is \ - visible; regenerate from the current sealed head", - batch.frame.cycle.0 - )), - Err(e) => WriteFailed(format!( - "TIMELINE ANOMALY, MANUAL RECONCILIATION REQUIRED: cycle {} \ - committed at DatasetVersion({published}) (expected {}), and the \ - compensating delete failed: {e}", - batch.frame.cycle.0, - base.0 + 1 - )), - }); - } - published - } - }; - Ok(DatasetVersion(published)) - } - - /// Committed landings only, in the STORED canonical order, from the - /// REOPENED dataset — never an in-memory echo. `from_version` filters to - /// cycles sealed strictly after it. - async fn scan_sealed( +impl LanceCycleWriter { + /// A sealed cycle's durable coalesced image: `row → final 512-byte + /// payload`. Projected `row` + `payload` under `kind = 2 AND cycle = …`. + pub async fn scan_image( &self, - from_version: Option, - ) -> Result, WriteFailed> { - let Some(ds) = self.open_if_exists().await? else { - return Ok(Vec::new()); - }; - let rows = self.read_rows_of_kind(&ds, KIND_LANDING).await?; - Ok(rows - .into_iter() - .filter(|r| from_version.is_none_or(|f| r.sealed_version() > f)) - .map(|r| LandedSlot { - version: r.sealed_version(), - slot: r.slot, - }) - .collect()) - } - - /// The cheap coarse timeline — the per-cycle frame rows, each sealed in the - /// same atomic commit as its landings, read back from the reopened store. - /// Projects only `cycle` + `base_version` under a `kind = 0` predicate, so - /// the lookup never materializes a single landing payload no matter how - /// much witness history the store has accumulated. - async fn versions(&self) -> Result, WriteFailed> { - let Some(ds) = self.open_if_exists().await? else { - return Ok(Vec::new()); + cycle: CycleId, + ) -> Result>, WriteFailed> { + let Some(ds) = self.ds.as_ref() else { + return Ok(std::collections::BTreeMap::new()); }; let mut scan = ds.scan(); scan.scan_in_order(true); - scan.filter(&format!("kind = {KIND_FRAME}")) - .map_err(|e| WriteFailed(format!("filter {}: {e}", self.dataset_path)))?; - scan.project(&["cycle", "base_version"]) - .map_err(|e| WriteFailed(format!("project {}: {e}", self.dataset_path)))?; + scan.filter(&format!("kind = {KIND_IMAGE} AND cycle = {}", cycle.0)) + .map_err(|e| WriteFailed(format!("filter: {e}")))?; + scan.project(&["row", "payload"]) + .map_err(|e| WriteFailed(format!("project: {e}")))?; let batches: Vec = scan .try_into_stream() .await - .map_err(|e| WriteFailed(format!("scan {}: {e}", self.dataset_path)))? + .map_err(|e| WriteFailed(format!("scan: {e}")))? .try_collect() .await - .map_err(|e| WriteFailed(format!("collect {}: {e}", self.dataset_path)))?; - let mut out = Vec::new(); + .map_err(|e| WriteFailed(format!("collect: {e}")))?; + let mut out = std::collections::BTreeMap::new(); for b in &batches { - let cycle: &UInt64Array = b - .column_by_name("cycle") + let row: &UInt64Array = b + .column_by_name("row") .and_then(|c| c.as_any().downcast_ref()) - .ok_or_else(|| WriteFailed("missing column cycle".into()))?; - let base_version: &UInt64Array = b - .column_by_name("base_version") + .ok_or_else(|| WriteFailed("missing column row".into()))?; + let payload: &FixedSizeBinaryArray = b + .column_by_name("payload") .and_then(|c| c.as_any().downcast_ref()) - .ok_or_else(|| WriteFailed("missing column base_version".into()))?; + .ok_or_else(|| WriteFailed("missing column payload".into()))?; for i in 0..b.num_rows() { - out.push(( - CycleId(cycle.value(i)), - DatasetVersion(base_version.value(i) + 1), - )); + out.insert(row.value(i), payload.value(i).to_vec()); } } Ok(out) @@ -617,8 +781,8 @@ impl WalSink for LanceCycleSink { } // --------------------------------------------------------------------------- -// Tests — every guarantee proven against a REOPENED dataset (fresh sink -// instance, fresh `Dataset::open`), never an in-memory echo. +// Falsifiers — every guarantee proven against a REOPENED store (a fresh +// `LanceCycleWriter::open` over the same path), never an in-memory echo. // --------------------------------------------------------------------------- #[cfg(test)] @@ -626,6 +790,10 @@ mod tests { use super::*; use lance_graph_planner::persist_sink::{persist_cycle, CycleFrame}; + fn witness(tag: u8) -> Vec { + vec![tag; EPISODIC_WITNESS_BYTES] + } + fn mv(owner: MailboxId) -> KanbanMove { KanbanMove { mailbox: owner, @@ -636,267 +804,543 @@ mod tests { } } - /// A canonical 512-byte witness payload, tagged by `stream_position` so - /// distinct casts stay byte-distinguishable. - fn witness(stream_position: u64) -> Vec { - vec![stream_position as u8; EPISODIC_WITNESS_BYTES] - } - - fn slot(cycle: u64, stream_position: u64, owner: MailboxId, row: u64) -> SweepSlot { + /// An ARTIFACT cast (non-empty canonical payload). + fn artifact(cycle: u64, sp: u64, owner: MailboxId, row: u64) -> SweepSlot { SweepSlot { cycle: CycleId(cycle), - stream_position, + stream_position: sp, owner, row, paired_move: Some(mv(owner)), - payload: witness(stream_position), + payload: witness(sp as u8), + } + } + + /// An INTENT-ONLY cast (empty payload — the `restage_held` shape). + fn intent(cycle: u64, sp: u64, owner: MailboxId, row: u64) -> SweepSlot { + SweepSlot { + payload: Vec::new(), + ..artifact(cycle, sp, owner, row) } } - /// One cycle → ONE official Lance commit → exactly one real DatasetVersion - /// `base + 1`; a fresh sink over the same path (restart) reads the sealed - /// landings and the cycle ↔ version mapping back from storage. + /// F1 + F13: zero artifact-backed delta → ZERO Lance operations, ZERO + /// version, and no dataset is even created; the store survives restart as + /// "empty", and a later real cycle still commits at V1. #[tokio::test] - async fn seal_survives_restart_and_reopen() { + async fn no_artifact_delta_writes_nothing_and_creates_no_version() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - - let frame = CycleFrame::new(CycleId(1), DatasetVersion(0)); - let casts = vec![slot(1, 20, 5, 100), slot(1, 10, 5, 101)]; - let v = persist_cycle(&sink, frame, casts).await.unwrap(); - assert_eq!(v, DatasetVersion(1)); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + assert_eq!(w.head(), DatasetVersion(0)); - // The REAL Lance version chain agrees — not a private counter. - let ds = Dataset::open(path.to_str().unwrap()).await.unwrap(); - assert_eq!(ds.version().version, 1); + // Thousands of pure kanban steps / held intents: nothing durable. + let intents: Vec = (0..2_000u64).map(|i| intent(1, i, 42, i % 7)).collect(); + let out = persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + intents, + ) + .await + .unwrap(); + assert_eq!( + out, + CommitOutcome::NoChange { + head: DatasetVersion(0) + } + ); + assert_eq!(w.head(), DatasetVersion(0), "no version was minted"); + assert!( + Dataset::open(path.to_str().unwrap()).await.is_err(), + "the dataset was never even created" + ); - // Restart: a brand-new sink instance, nothing shared but the path. - let reopened = LanceCycleSink::new(path.to_str().unwrap()); - let sealed = reopened.scan_sealed(None).await.unwrap(); - assert_eq!(sealed.len(), 2); - // Stored canonical order (deinterlaced at freeze: 10 before 20) — the - // scan preserves it, it does not repair it. - assert_eq!(sealed[0].slot.stream_position, 10); - assert_eq!(sealed[1].slot.stream_position, 20); - assert_eq!(sealed[0].version, DatasetVersion(1)); - assert_eq!(sealed[0].slot.paired_move, Some(mv(5))); - assert_eq!(sealed[0].slot.payload, witness(10)); - - let versions = reopened.versions().await.unwrap(); - assert_eq!(versions, vec![(CycleId(1), DatasetVersion(1))]); + // Restart: still empty, and a real artifact cycle commits at V1. + drop(w); // the registry enforces one live writer per path + let mut w2 = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + assert!(w2.timeline().await.unwrap().is_empty()); + let out = persist_cycle( + &mut w2, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![artifact(1, 0, 42, 5)], + ) + .await + .unwrap(); + assert!( + matches!( + out, + CommitOutcome::Committed { + version: DatasetVersion(1), + .. + } + ), + "{out:?}" + ); } - /// A stale `base` is fenced with NOTHING written: the store's version chain, - /// landings, and timeline are untouched — proven on reopen. + /// F2 + the measured bytes-written falsifier: 64 transient breaths on ONE + /// row cost exactly ONE 512-byte durable image row — not 64 × 512. #[tokio::test] - async fn stale_base_is_fenced_and_writes_nothing() { + async fn sixty_four_breaths_on_one_row_cost_one_image_row() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - - // Empty store: a caller claiming a sealed predecessor V3 is refused. - let err = sink - .commit_cycle( - DatasetVersion(3), - DetachedCycleBatch::freeze(CycleFrame::new(CycleId(1), DatasetVersion(3)), vec![]), - ) + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) .await - .unwrap_err(); - assert!(err.0.contains("stale base"), "{err}"); - assert!(Dataset::open(path.to_str().unwrap()).await.is_err()); + .unwrap(); - // Seal cycle 1 at base 0 → V1; then a sibling still reading base 0 is - // fenced, and the store is byte-for-byte the sealed head it was. + // One thought, 64 successive artifact updates to the SAME row. + let casts: Vec = (0..64u64).map(|i| artifact(1, i, 42, 9)).collect(); persist_cycle( - &sink, + &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), - vec![slot(1, 1, 2, 40)], + casts, ) .await .unwrap(); - let err = sink - .commit_cycle( - DatasetVersion(0), - DetachedCycleBatch::freeze( - CycleFrame::new(CycleId(2), DatasetVersion(0)), - vec![slot(2, 2, 2, 41)], - ), - ) - .await - .unwrap_err(); - assert!(err.0.contains("stale base"), "{err}"); - let reopened = LanceCycleSink::new(path.to_str().unwrap()); - let ds = Dataset::open(path.to_str().unwrap()).await.unwrap(); - assert_eq!(ds.version().version, 1, "fenced commit must not publish"); - assert_eq!(reopened.scan_sealed(None).await.unwrap().len(), 1); - assert_eq!(reopened.versions().await.unwrap().len(), 1); + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let image = reopened.scan_image(CycleId(1)).await.unwrap(); + assert_eq!(image.len(), 1, "exactly ONE durable row image"); + assert_eq!( + image[&9], + witness(63), + "the LAST breath survived (later stream position wins)" + ); + let durable_payload_bytes: usize = image.values().map(Vec::len).sum(); + assert_eq!( + durable_payload_bytes, + EPISODIC_WITNESS_BYTES, + "512 durable payload bytes, not 64 x 512 = {}", + 64 * EPISODIC_WITNESS_BYTES + ); } - /// Sequential cycles chain the sealed horizon: V1 → V2 → V3; `scan_sealed` - /// filters strictly-after; `versions` is the full coarse timeline. + /// F3-adjacent + F10: a successful commit performs ZERO reopens, and the + /// caller references the returned version instead of reloading state. #[tokio::test] - async fn sequential_cycles_chain_and_filter() { + async fn successful_commit_reopens_nothing() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - - for (cycle, base) in [(1u64, 0u64), (2, 1), (3, 2)] { - let v = persist_cycle( - &sink, - CycleFrame::new(CycleId(cycle), DatasetVersion(base)), - vec![slot(cycle, cycle * 10, 9, cycle)], + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let opens_after_startup = w.opens(); + + for c in 1..=3u64 { + let out = persist_cycle( + &mut w, + CycleFrame::new(CycleId(c), DatasetVersion(c - 1)), + vec![artifact(c, c, 42, c)], ) .await .unwrap(); - assert_eq!(v, DatasetVersion(base + 1)); + assert!(matches!(out, CommitOutcome::Committed { .. })); } - - let reopened = LanceCycleSink::new(path.to_str().unwrap()); assert_eq!( - reopened.versions().await.unwrap(), - vec![ - (CycleId(1), DatasetVersion(1)), - (CycleId(2), DatasetVersion(2)), - (CycleId(3), DatasetVersion(3)), - ] + w.opens(), + opens_after_startup, + "no reopen on the normal commit path" + ); + assert_eq!( + w.reconcile_scans(), + 0, + "ZERO reconciliation scans on the fresh-monotonic normal path — measured, not asserted" + ); + assert_eq!( + w.head(), + DatasetVersion(3), + "the head token tracks in memory" ); - // Strictly after V1: cycles 2 and 3 only. - let after_v1 = reopened.scan_sealed(Some(DatasetVersion(1))).await.unwrap(); - assert_eq!(after_v1.len(), 2); - assert!(after_v1.iter().all(|l| l.version > DatasetVersion(1))); } - /// A zero-landing cycle still advances the sealed timeline (its frame row - /// commits atomically) while contributing nothing to `scan_sealed`. + /// The one-writer topology is ENFORCED in-process: a second live writer on + /// the same path is refused at open; dropping the first frees the path. + /// (Cross-process exclusivity remains a deployment lease — documented.) #[tokio::test] - async fn empty_cycle_advances_timeline_only() { + async fn a_second_live_writer_on_the_same_path_is_refused() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); + let path = dir.path().join("cycles.lance"); + let w1 = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let second = LanceCycleWriter::open(path.to_str().unwrap()).await; + assert!( + second.is_err(), + "two live writers over one store must be unrepresentable in-process" + ); + drop(w1); + let w3 = LanceCycleWriter::open(path.to_str().unwrap()).await; + assert!(w3.is_ok(), "dropping the writer frees the path"); + } - let v = persist_cycle( - &sink, + /// Restart reconciliation stays cheap AND correct: a re-submitted batch + /// after restart reconciles (one scan), while fresh cycles keep the + /// scan-free path. + #[tokio::test] + async fn restart_resubmission_reconciles_with_one_scan() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + { + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![artifact(1, 0, 42, 1)], + ) + .await + .unwrap(); + } + // Restart (lost acknowledgement): same frozen batch re-submitted. + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let retry = persist_cycle( + &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), - vec![], + vec![artifact(1, 0, 42, 1)], + ) + .await + .unwrap(); + assert!( + matches!(retry, CommitOutcome::Reconciled { .. }), + "{retry:?}" + ); + assert_eq!(w.reconcile_scans(), 1, "exactly one reconciliation scan"); + // A fresh cycle afterwards is scan-free again. + persist_cycle( + &mut w, + CycleFrame::new(CycleId(2), DatasetVersion(1)), + vec![artifact(2, 1, 42, 2)], ) .await .unwrap(); - assert_eq!(v, DatasetVersion(1)); + assert_eq!(w.reconcile_scans(), 1, "the fresh cycle added no scan"); + } - let reopened = LanceCycleSink::new(path.to_str().unwrap()); - assert!(reopened.scan_sealed(None).await.unwrap().is_empty()); + /// F12 + F19: the recovery tail is BOUNDED — `after_cycle` excludes earlier + /// history, and landing reads never touch the payload column. + #[tokio::test] + async fn bounded_tail_recovery_reads_no_payloads() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + for c in 1..=4u64 { + persist_cycle( + &mut w, + CycleFrame::new(CycleId(c), DatasetVersion(c - 1)), + vec![artifact(c, c, 42, c)], + ) + .await + .unwrap(); + } + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let tail = reopened.scan_sealed(Some(CycleId(2))).await.unwrap(); assert_eq!( - reopened.versions().await.unwrap(), - vec![(CycleId(1), DatasetVersion(1))] + tail.iter().map(|l| l.cycle).collect::>(), + vec![CycleId(3), CycleId(4)], + "strictly after the bound" + ); + assert!( + tail.iter().all(|l| l.slot.payload.is_empty()), + "landing reads carry no payload — the column was never projected" + ); + assert!( + tail.iter().all(|l| l.slot.paired_move.is_some()), + "but the transition metadata IS there (recovery needs the moves)" ); } - /// An empty store is a state, not an error: nothing sealed, empty timeline. + /// F11: the timeline is frame-only and payload-free, and survives restart. #[tokio::test] - async fn empty_store_reads_empty() { + async fn timeline_is_frame_metadata_only() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("never_created.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - assert!(sink.scan_sealed(None).await.unwrap().is_empty()); - assert!(sink.versions().await.unwrap().is_empty()); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + for c in 1..=2u64 { + persist_cycle( + &mut w, + CycleFrame::new(CycleId(c), DatasetVersion(c - 1)), + vec![artifact(c, c, 42, c)], + ) + .await + .unwrap(); + } + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let frames = reopened.timeline().await.unwrap(); + assert_eq!( + frames + .iter() + .map(|f| (f.cycle, f.base_version)) + .collect::>(), + vec![ + (CycleId(1), DatasetVersion(0)), + (CycleId(2), DatasetVersion(1)), + ] + ); + assert!(frames.iter().all(|f| f.batch_hash != 0)); } - /// A no-move landing round-trips as `None` (nullable move columns), and a - /// large-ish payload survives byte-exact. + /// F10 + F12 (the reconciliation half): re-submitting the SAME frozen batch + /// after a "lost acknowledgement" reconciles to exactly one conclusion — no + /// duplicate rows, no second version, and NO delete anywhere. #[tokio::test] - async fn move_nullability_and_payload_roundtrip() { + async fn resubmitting_the_same_batch_reconciles_to_one() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let casts = || vec![artifact(1, 0, 42, 1), artifact(1, 1, 42, 2)]; - let witness_node = (0..=255u8).cycle().take(512).collect::>(); - let mut s = slot(1, 3, 11, 900); - s.paired_move = None; - s.payload = witness_node.clone(); - persist_cycle( - &sink, + let first = persist_cycle( + &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), - vec![s], + casts(), ) .await .unwrap(); + assert!(matches!( + first, + CommitOutcome::Committed { + version: DatasetVersion(1), + .. + } + )); - let reopened = LanceCycleSink::new(path.to_str().unwrap()); - let sealed = reopened.scan_sealed(None).await.unwrap(); - assert_eq!(sealed.len(), 1); - assert_eq!(sealed[0].slot.paired_move, None); - assert_eq!(sealed[0].slot.payload, witness_node); - assert_eq!(sealed[0].slot.owner, 11); - assert_eq!(sealed[0].slot.row, 900); + // The response was lost; the caller retries the identical batch. + let retry = persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + casts(), + ) + .await + .unwrap(); + assert!( + matches!( + retry, + CommitOutcome::Reconciled { + cycle: CycleId(1), + .. + } + ), + "{retry:?}" + ); + assert_eq!(w.head(), DatasetVersion(1), "no second version"); + + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + assert_eq!( + reopened.scan_sealed(None).await.unwrap().len(), + 2, + "no duplicate landings after restart" + ); + assert_eq!(reopened.timeline().await.unwrap().len(), 1, "one frame"); } - /// A malformed witness payload (≠ 512 bytes) is refused with NOTHING - /// written — the canonical node row stride is enforced before anything - /// durable happens, and the store stays exactly as it was. + /// F10 (the fail-closed half): a DIFFERENT batch for a durable cycle is + /// refused loudly and writes nothing. #[tokio::test] - async fn malformed_payload_is_refused_before_persistence() { + async fn a_conflicting_batch_for_a_durable_cycle_fails_closed() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - - let mut bad = slot(1, 1, 3, 50); - bad.payload = vec![0u8; 100]; - let err = persist_cycle( - &sink, + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + persist_cycle( + &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), - vec![bad], + vec![artifact(1, 0, 42, 1)], ) .await - .unwrap_err(); - assert!(err.to_string().contains("100 bytes"), "{err}"); - // Nothing durable: the dataset was never even created. - assert!(Dataset::open(path.to_str().unwrap()).await.is_err()); + .unwrap(); + let conflict = persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![artifact(1, 9, 42, 1)], + ) + .await; + assert!( + matches!( + conflict, + Err(lance_graph_planner::persist_sink::PersistError::Commit( + CommitError::HashConflict { + cycle: CycleId(1), + .. + } + )) + ), + "{conflict:?}" + ); + assert_eq!(w.head(), DatasetVersion(1), "nothing was written"); + } - // The store still accepts a well-formed cycle afterwards. + /// F11-adjacent: a stale horizon is FENCED with the current head and + /// writes nothing (never a delete, never a silent accept). + #[tokio::test] + async fn a_stale_horizon_is_fenced_and_writes_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); persist_cycle( - &sink, + &mut w, CycleFrame::new(CycleId(1), DatasetVersion(0)), - vec![slot(1, 1, 3, 50)], + vec![artifact(1, 0, 42, 1)], ) .await .unwrap(); + let stale = persist_cycle( + &mut w, + CycleFrame::new(CycleId(2), DatasetVersion(0)), + vec![artifact(2, 1, 42, 2)], + ) + .await; + assert!( + matches!( + stale, + Err(lance_graph_planner::persist_sink::PersistError::Commit( + CommitError::Fenced { + current_head: DatasetVersion(1) + } + )) + ), + "{stale:?}" + ); + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + assert_eq!(reopened.timeline().await.unwrap().len(), 1); + assert_eq!(reopened.scan_sealed(None).await.unwrap().len(), 1); } - /// The coalesced image is DURABLE: same-row casts fold to the final - /// payload, persisted as image rows in the same atomic commit and read - /// back per cycle from a reopened store — while the per-cast landing - /// history stays intact alongside it. + /// F5 + F6: no persisted row carries a nonterminal-only cast. Intent-only + /// casts (held work, pure `Continue`-style movement) leave NO trace, while + /// artifact casts in the same cycle persist normally. #[tokio::test] - async fn coalesced_image_is_durable_per_cycle() { + async fn intent_only_casts_leave_no_durable_trace() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("witness_cycles.lance"); - let sink = LanceCycleSink::new(path.to_str().unwrap()); - - // Three casts, two rows: row 7 is written twice (positions 1 then 3 — - // later stream position wins the image), row 8 once. - let casts = vec![slot(1, 3, 2, 7), slot(1, 1, 2, 7), slot(1, 2, 2, 8)]; - persist_cycle(&sink, CycleFrame::new(CycleId(1), DatasetVersion(0)), casts) + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) .await .unwrap(); + let mixed = vec![ + intent(1, 0, 42, 1), + artifact(1, 1, 42, 2), + intent(1, 2, 99, 3), + intent(1, 3, 99, 4), + ]; + persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + mixed, + ) + .await + .unwrap(); - let reopened = LanceCycleSink::new(path.to_str().unwrap()); - let image = reopened.scan_image(CycleId(1)).await.unwrap(); - assert_eq!(image.len(), 2); - assert_eq!(image[&7], witness(3), "later stream position wins"); - assert_eq!(image[&8], witness(2)); - // The per-cast history is still complete and ordered. + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); let sealed = reopened.scan_sealed(None).await.unwrap(); - assert_eq!(sealed.len(), 3); + assert_eq!(sealed.len(), 1, "only the artifact cast persisted"); + assert_eq!(sealed[0].slot.stream_position, 1); + assert_eq!(reopened.scan_image(CycleId(1)).await.unwrap().len(), 1); + } + + /// F15: randomized completion order yields the same durable result — + /// identical batch hash, identical landings, identical image. + #[tokio::test] + async fn randomized_completion_order_yields_the_same_durable_set() { + let ordered = vec![ + artifact(1, 0, 1, 10), + artifact(1, 1, 2, 11), + artifact(1, 2, 3, 12), + ]; + let scrambled = vec![ + artifact(1, 2, 3, 12), + artifact(1, 0, 1, 10), + artifact(1, 1, 2, 11), + ]; + + let mut hashes = Vec::new(); + let mut images = Vec::new(); + for casts in [ordered, scrambled] { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + casts, + ) + .await + .unwrap(); + drop(w); // a restart means the prior writer is GONE (the registry enforces it) + let reopened = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + hashes.push(reopened.timeline().await.unwrap()[0].batch_hash); + let sealed = reopened.scan_sealed(None).await.unwrap(); + assert_eq!( + sealed + .iter() + .map(|l| l.slot.stream_position) + .collect::>(), + vec![0, 1, 2], + "stored in canonical order regardless of arrival" + ); + images.push(reopened.scan_image(CycleId(1)).await.unwrap()); + } assert_eq!( - sealed - .iter() - .map(|l| l.slot.stream_position) - .collect::>(), - vec![1, 2, 3] + hashes[0], hashes[1], + "same conclusion set → same batch hash" ); - // An unknown cycle has no image. - assert!(reopened.scan_image(CycleId(99)).await.unwrap().is_empty()); + assert_eq!(images[0], images[1], "same durable image"); + } + + /// The ABI gate bites on an ARTIFACT payload (and cannot bite on an + /// intent-only cast, which never reaches the writer). + #[tokio::test] + async fn a_malformed_artifact_payload_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let mut w = LanceCycleWriter::open(path.to_str().unwrap()) + .await + .unwrap(); + let mut bad = artifact(1, 0, 42, 1); + bad.payload = vec![7u8; 511]; + let r = persist_cycle( + &mut w, + CycleFrame::new(CycleId(1), DatasetVersion(0)), + vec![bad], + ) + .await; + assert!(r.is_err(), "511 bytes must be refused: {r:?}"); + assert_eq!(w.head(), DatasetVersion(0), "nothing written"); } }