Phase A: artifact-backed commits + the sole owned Lance writer (supersedes the #911 cycle contract)

.claude/board/EPIPHANIES.md

@@ -1,3 +1,69 @@
+## 2026-08-09
+
+### E-THE-ARTIFACT-WRITE-DECIDES-WHAT-KANBAN-PROGRESS-BECOMES-DURABLE-1
+
+**FINDING (operator-ruled, implemented Phase A).** The persistence question was
+being asked backwards. The old contract let the *cycle* decide when to write —
+every 550 ms sweep sealed a `DatasetVersion`, so a thought crossing dozens of
+cheap Rubicon steps in ~2 s minted dozens of versions and stretched a ~2 s
+ladder across ~32-35 s of barriers. The correct rule inverts the initiative:
+
+> **Kanban never decides when to persist. The semantic write that happens
+> anyway decides which kanban progress gets a durable anchor.**
+
+Thinking is cheaper than persisting its intermediate control flow. Rubicon
+state, `Continue`, `Hold`, held work, the current rung and scheduler progress
+stay **transient** — after a crash they are cheaply regenerated from the pinned
+sealed inputs, which is exactly the property that makes discarding them safe.
+
+**The mechanical form is smaller than the ruling sounds.** The gate already
+existed in the data: a cast's payload. Non-empty = an artifact (grounding
+result, reusable walk product, adjudication, conclusion) → persisted.
+Empty = intent-only → ephemeral. `restage_held` had been casting empty payloads
+since #879 (`cycle_driver.rs:303`), and #911's 512-byte ABI gate contradicted
+it — the *fix* was not padding those casts to 512 bytes to satisfy the gate but
+recognizing that **the empty payload IS the ephemerality mechanism**. Two
+post-merge P1s dissolved at once: intent-only casts can never trip a payload
+gate they never reach, and the empty-cycle version disappears because zero
+artifact casts means the sink is never called at all.
+
+**The general lesson.** When a gate and a producer contradict each other, the
+question "which one do I bend?" often has a third answer: the contradiction is
+the system telling you the two things were never in the same category. A
+lifecycle transition and a semantic artifact are not the same kind of fact, and
+only one of them belongs in durable storage.
+
+### E-A-PUBLISHED-MANIFEST-IS-HISTORY-RECONCILIATION-NOT-ROLLBACK-1
+
+**FINDING (measured against `lance-9.0.0/src/io/commit.rs:914-950`).** #911
+tried to make an optimistic fence *effective* after the fact: on detecting that
+a foreign writer had shifted the published version, it issued a compensating
+`Dataset::delete` scoped to the cycle, then returned a retryable error. Review
+found the flaw (the `(cycle, base_version)` predicate can delete a *successful
+concurrent writer's* rows — the very race it exists to handle), but the deeper
+error is categorical: **`Dataset::delete` creates another version. It is not
+rollback.** There is no undo in an MVCC manifest chain.
+
+The measured constraint underneath: **Lance 9 has no atomic expected-version
+fence for `Append`** — the conflict rebase runs even on a single-attempt commit;
+strict no-rebase mode exists only for `Overwrite`. A read-then-append is not a
+compare-and-swap, and dressing it as one produces exactly the
+committed-but-reported-failed hole that made the driver regenerate work that
+had already landed.
+
+The resolution is to stop trying to make the fence retroactive and make
+**reconciliation authoritative** instead: commit the batch's identity
+`(cycle, batch_hash)` *in the same commit as its rows*, look it up before
+appending, and let a lost acknowledgement resolve by re-submitting the SAME
+frozen batch. Same identity + same hash ⇒ `Reconciled` (success, no second
+append). Same identity + different hash ⇒ fail closed. Genuinely unknown ⇒ an
+`Ambiguous` state that says so, instead of an error that falsely promises
+nothing landed.
+
+**The rule this leaves behind:** never return an error meaning "nothing
+landed" when failure could have occurred after publication — and never delete
+history to restore an expected version number.
+
 ## 2026-08-07
 
 ### E-COMPRESSION-META-INERT-AT-512-STRIDE-1

.claude/board/INTEGRATION_PLANS.md

@@ -1,3 +1,14 @@
+## 2026-08-09 — persistence-artifact-backed-commit v1 — RATIFIED (Phase A implemented; B–F planned) — main thread
+
+The canonical persistence contract, replacing the #911 cycle-persistence model.
+Plan: `.claude/plans/persistence-artifact-backed-commit-v1.md`.
+
+- **The rule:** no artifact-backed semantic change → no write → no new `DatasetVersion`. Thinking runs its whole Rubicon ladder transiently; only a semantic artifact becomes durable, and kanban progress rides along in the commit that was happening anyway.
+- **One logical writer**, split by capability: `Clone + &self` for producer submission / read-only projections; the concrete `LanceCycleWriter` non-`Clone`, owning its `Dataset` handle + head, committing through `&mut self`.
+- **No rollback, no compensating delete** — a published manifest is history (measured: Lance 9 has no atomic expected-version Append fence); durable in-band `(cycle, batch_hash)` makes reconciliation authoritative.
+- **Zero reload on the normal path**, instrumented by `opens()`; reads bounded (`after_cycle`) and projected (timeline never touches payloads).
+- **Phases:** A (this contract + owned writer) — implemented on `claude/phase-a-owned-writer`; B shared representation/projection ABI; C live granular visibility + wavefront; D conclusion boundary; E MedCare proof + Gotham wiring; F A2UI/ClassView renderer. Each phase restarts from merged main; public lance-graph stays generic, clinical mappings stay private to MedCare-rs.
+
 ## 2026-08-06 — idle-flush-dataset-eviction v1 — PROPOSAL (not scheduled; nothing implemented, nothing measured)
 
 **Plan:** `.claude/plans/idle-flush-dataset-eviction-v1.md`

.claude/board/LATEST_STATE.md

@@ -1,3 +1,16 @@
+## 2026-08-09 — branch `claude/phase-a-owned-writer` — Phase A: the artifact-backed commit contract + the SOLE owned Lance writer (`LanceCycleWriter`)
+
+> **⊘ This entry SUPERSEDES the #911 entry below it** (operator ruling 2026-08-09). The §I.6 "every cycle publishes exactly one `DatasetVersion`" contract, the compensating delete, and the per-operation-reopen sink are all REMOVED — not repaired. Canonical record: `.claude/plans/persistence-artifact-backed-commit-v1.md`. Nothing below is deleted; it is read through that document.
+
+### Current Contract Inventory — reshaped (lance-graph-planner) + replaced (lance-graph core, `planner` feature)
+
+- **The governing storage rule** (`persist_sink`): **no artifact-backed semantic change → no write → no new `DatasetVersion`.** Thinking is cheaper than persisting its intermediate control flow: a thought runs its whole Rubicon ladder transiently and only an artifact-backed delta becomes durable. Mechanically the gate is the cast payload — NON-EMPTY = artifact cast (persisted), EMPTY = intent-only cast (held-intent re-stage / pure kanban step) which `persist_cycle` partitions out as **ephemeral**. Zero artifact casts ⇒ `CommitOutcome::NoChange { head }` with the sink NEVER called (zero store ops, zero rows, unchanged version). `restage_held`'s empty payload is thereby exactly what makes a re-staged intent ephemeral — no ABI gate can trip on it, because such a cast never reaches the writer. #911's deliberate empty-cycle versioning is REMOVED and its falsifier inverted.
+- **`WalSink` reshaped**: `commit_cycle(&mut self, batch) -> Result<CommitOutcome, CommitError>` (the `base` param is gone — it lives in `batch.frame`), `scan_sealed(after_cycle: Option<CycleId>)` (cycle-bounded tail, pushed into the scan), `timeline() -> Vec<FrameMeta>` (replaces `versions()`; frame metadata only, never a payload). `LandedSlot` is now `{ cycle, slot }` — the derived `version` field is GONE (a physical publication position is not a per-row semantic identity). `DetachedCycleBatch` gained `batch_hash` (FNV-1a 64 over the CANONICAL content, so randomized completion order yields an identical hash).
+- **Honest commit states**: `NoChange { head }` / `Committed { version, cycle, batch_hash }` / `Reconciled { version, cycle, batch_hash }`; errors `Fenced { current_head }` / `HashConflict { cycle, stored_hash, offered_hash }` / `Io(WriteFailed)` / `Ambiguous { cycle, batch_hash, cause }`. **No error may promise "nothing landed" when failure could have occurred after manifest publication** — that case reconciles or surfaces as `Ambiguous`.
+- **`LanceCycleWriter`** (replaces `LanceCycleSink`) — the SOLE application writer: **non-`Clone`**, owns a **long-lived `Dataset` handle + in-memory head**, commits through **`&mut self`**. The capability split is deliberate (operator correction): `Clone + &self` belongs to producer submission and read-only projections; fire-and-forget means producers get no acknowledgement, NOT that the writer ignores the result. **No rollback, no compensating delete** — a published manifest is history; `Dataset::delete` mints another version and is not rollback (and #911's `(cycle, base_version)` predicate could destroy a concurrent same-cycle winner). Idempotency is durable and in-band: `(cycle, batch_hash)` committed with the rows, reconciled FIRST, so re-submitting the same frozen batch after a lost acknowledgement returns `Reconciled` instead of double-appending. Lance 9 has NO atomic expected-version fence for Append (rebase runs even single-attempt; strict mode is Overwrite-only — measured in `lance-9.0.0/src/io/commit.rs:914-950`); that is stated, not papered over.
+- **Zero reload on the normal path, instrumented**: `LanceCycleWriter::opens()` counts every `Dataset::open` ever performed (startup + ambiguity resolution only). Layout is three row kinds — frame (1/cycle), landing metadata (1/artifact cast, payload NULL), coalesced image (1 per DIRTY ROW, the final 512-byte payload). Payload is physically `FixedSizeBinary(512)`; reads are bounded + projected (`timeline` never scans the payload column; `scan_sealed` returns transition metadata only; `scan_image` projects payload on request).
+- **Gates:** 11 reopened-store falsifiers in `cycle_sink.rs` + 5 contract falsifiers in `persist_sink.rs`, including the **measured bytes-written** one — 64 transient breaths on one row cost **512 durable bytes, not 64 × 512**. Honestly deferred (named, not skipped): the real object-store RUN — S3 needs lance's `aws` feature, which our `lance = "=9.0.0"` default-features pin already enables (verified: `aws-config`/`aws-credential-types` in the graph), so an `s3://` store compiles and routes today; what is unmeasured is the credentialed commit/reconciliation/tail-read, and **no object-store durability claim is made** until it runs. Also deferred: `Continue`-from-pulse (Phase C), 64k-scale measurement.
+
 ## 2026-08-09 — branch `claude/medcare-rs-continue-ufsazd` — `lance_graph::graph::cycle_sink`: the CONCRETE cognitive-cycle Lance sink (the storage-proven `WalSink`)
 
 ### Current Contract Inventory — new module (lance-graph core, `planner` feature)

.claude/board/PR_ARC_INVENTORY.md

@@ -33,14 +33,24 @@
 > - **Docs** — knowledge files produced (immutable)
 > - **Confidence (YYYY-MM-DD):** — the ONLY mutable field
 
+## 2026-08-09 — branch `claude/phase-a-owned-writer` (PR pending) — Phase A: the artifact-backed commit contract + the SOLE owned writer
+
+- **Added.** `.claude/plans/persistence-artifact-backed-commit-v1.md` (the canonical persistence contract, ratified). `lance_graph::graph::cycle_sink::LanceCycleWriter` (~900 LOC incl. 11 reopened-store falsifiers) replacing `LanceCycleSink`. In `persist_sink`: `CommitOutcome` / `CommitError` / `FrameMeta`, `DetachedCycleBatch::batch_hash`, the reshaped `WalSink` (`commit_cycle(&mut self, batch)`, `scan_sealed(after_cycle)`, `timeline()`), `PersistError::Commit`, and 5 contract falsifiers.
+- **Locked.** **No artifact-backed semantic change → no write → no new DatasetVersion** (empty/intent-only cycles perform ZERO store operations; `restage_held`'s empty payload IS the ephemerality mechanism). **One logical writer, split by CAPABILITY** — `Clone + &self` for producer submission and read-only projections; the concrete writer non-`Clone`, owning its `Dataset` handle + head, committing through `&mut self`; fire-and-forget means producers get no acknowledgement, never that the writer ignores the result. **No rollback and no compensating delete** — a published manifest is history; idempotency is durable in-band `(cycle, batch_hash)`, reconciled first, so a lost acknowledgement is resolved by re-submitting the same frozen batch. **No error may promise "nothing landed" after possible publication** (`Ambiguous` is the honest unknown). **Zero reload on the normal path**, instrumented via `opens()`. **Lance 9 has no atomic expected-version Append fence** — measured at `lance-9.0.0/src/io/commit.rs:914-950`, stated rather than papered over. **`DatasetVersion` is a physical publication position**, not a per-row semantic identity (`LandedSlot.version` removed).
+- **Deferred.** Real object-store commit + ambiguous-response paths (no credentials in this environment — NO object-store durability claim is made). True zero-copy: the copy boundary (Arrow builder materialization + `to_vec` readback) is documented and isolated for a later measured PR, along with the `BatchWriter<P>`-descriptor-vs-`Vec<u8>` contradiction. Phases B–F (representation/projection ABI, live pulse + wavefront, conclusion boundary, MedCare proof, A2UI renderer).
+- **Docs.** The canonical plan above; a `⊘ PARTIALLY SUPERSEDED` header on `persistence-cycle-wal-bootstrap-v1.md` (guarantee 5 now conditional; guarantees 1–4/6 survive, §2 sparse-delta now IMPLEMENTED); LATEST_STATE entry marking the #911 entry superseded.
+
+**Confidence (2026-08-09):** contract + writer falsifiers green against reopened local stores; not yet merged; object-store unproven by design.
+**Correction (2026-08-09, review round on #912):** five review findings fixed in place — (1) `reopen` can no longer degrade an existing store to empty (a held handle survives a transient NotFound; outcome stays Ambiguous); (2) the in-process one-writer topology is now ENFORCED via a process-local path registry (second live `open` refused; Drop frees), cross-process stays a documented deployment lease; (3) the normal commit path is now genuinely scan-free (cycle watermark seeded at open; `reconcile_scans()` instrumented and falsified at 0 for fresh monotonic commits; reconciliation only on fence-fail / re-submission / ambiguity); (4) `Reconciled.version` renamed `current_head` — it is NOT a publication version; only `Committed.version` is audit-grade; (5) `recover_fleet` now reports `foreign_landings` + `foreign_min_cycle` (the latecomer fence: never raise the global bound past it). Honestly recorded, not fixed: the artifact gate tests payload PRESENCE not semantic CHANGE (Phase-D conclusion-identity refinement, documented in the module doc); the 64-breaths falsifier now states the 64 compact landing-metadata rows explicitly (per-cast metadata collapse = the Phase B/C KanbanRollup); `run_cycle`'s doc no longer claims the fleet borrow is released across the await — it names the split (`collect → seal_cycle → apply`) as the production detached path.
+
 ## 2026-08-09 — branch `claude/medcare-rs-continue-ufsazd` (PR pending) — the concrete cognitive-cycle Lance sink: `graph::cycle_sink::LanceCycleSink`
 
 - **Added.** `lance_graph::graph::cycle_sink` (~660 LOC incl. 6 reopened-dataset tokio tests) — the concrete `lance_graph_planner::persist_sink::WalSink` over the official Lance 9 insert path; `cycle_store_schema()` (frame row + landing rows, nullable Rubicon `move_*` columns, `payload` witness bytes); `LanceCycleSink`. Module gated on the default-on `planner` feature.
 - **Locked.** **Lance's manifest/version chain IS the WAL** — no bespoke ledger / acknowledgement protocol / parallel replay system (operator-ruled). **The §I.6 fence has two halves**: pre-commit head==base (nothing written on refusal) AND post-commit published==base+1 — and because Lance has NO expected-version conditional append (Append rebases even single-attempt; strict mode is Overwrite-only, measured in `lance-9.0.0/src/io/commit.rs`), a detected shift triggers a compensating official `Dataset::delete` of exactly this cycle's rows BEFORE the retryable `WriteFailed` returns (review round, #911 Codex P1): the fence is effective retroactively, nothing of the cycle stays visible, regenerate-from-`Vn` stays sound. **The coalesced image is durable** (`kind=2` rows + `scan_image`, same atomic commit); **payloads are gated to the canonical 512 bytes**; **`versions()` is a projected frame-only read**. **`sealed_version = base_version + 1` is a verified identity, not an assumption** — which is what lets reads derive the cycle↔version mapping from the atomically-co-committed frame row with zero sidecar state. **Order is a write-side property** (stored deinterlaced order, `scan_in_order(true)`, no read-time sort). **Domain-0x09 witness contract**: the patient SoA is the ONLY patient-reasoning Lance write target; payload = maximally-rich EpisodicWitness node built ON TOP of the immutable domain-0x03 ontology addresses (immutability taken for granted for the representation window via `base_version`).
 - **Deferred.** The MedCare consumer arc (production `drive_cohort_thoughts` caller, witness-seal, views reading the sealed version) — next PR, in MedCare-rs. `recover_and_apply` wiring against this sink in a production driver. Object-store (s3/az/gs) smoke — the path plumbing accepts URIs but only local was exercised.
 - **Docs.** Module-level witness/§I.6 contract in `cycle_sink.rs`; this entry + LATEST_STATE inventory (same commit).
 
-**Confidence (2026-08-09):** tests green against reopened local datasets; not yet merged.
+**Confidence (2026-08-09):** MERGED as `8a5be50`. **SUPERSEDED the same day by the Phase-A entry above** (operator ruling): the §I.6 one-version-per-cycle contract, the compensating delete, and the per-operation reopen are removed — the post-merge review also confirmed two P1s in this entry's shape (the delete's `(cycle, base_version)` predicate can destroy a concurrent same-cycle winner; the 512-byte gate contradicts `restage_held`'s intent-only empty payload). Both are resolved structurally in Phase A rather than patched here.
 
 ## 2026-08-05 — the lance 9 / DataFusion 54 / Rust 1.97.1 cross-repo bump (9 repos; lance-graph PR pending, siblings MERGED)